KEY POINT CORRECTION DEVICE, KEY POINT CORRECTION METHOD, AND PROGRAM

Abstract

To improve accuracy of a key point of a mobile object detected from a photographed image.SOLUTION: A key point correction device 2000 detects a plurality of key points of a target object 20 from each of a plurality of photographed images 10. The key point correction device 2000 determines whether or not the plurality of key points 30 of the target object 20 detected from the photographed images 10 satisfy an appropriate condition. When the key point 30 of the target object 20 satisfies the appropriate condition, the key point correction device 2000 decides a correction parameter for the target object 20 by using the key point 30. When the correction parameter is decided for the target object 20 detected from the photographed image 10, the key point correction device 2000 corrects the key point 30 of the target object 20 by using the correction parameter.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a technique for detecting key points of a moving object included in a captured image. [Background technology]

[0002] Technologies have been developed for detecting key points on images of moving objects such as vehicles and people by analyzing captured images obtained by capturing the images of the moving objects. For example, Patent Document 1 discloses a technology for detecting key points of a moving object from multiple captured images generated at different times, and stably tracking a moving object that partially deforms by using the detected key points. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-039617 A Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 does not mention the accuracy of keypoints. The present disclosure provides a technique for improving the accuracy of keypoints of a moving object detected from a captured image. [Means for solving the problem]

[0005] The key point correction device disclosed herein includes an acquisition unit that acquires multiple captured images, a detection unit that detects multiple key points of a moving object from the captured images, a determination unit that determines whether the multiple key points of the moving object detected from the captured images satisfy specified suitability conditions, a determination unit that, when it is determined that the multiple key points of the moving object satisfy the suitability conditions, determines correction parameters to be used for correcting the multiple key points detected for the moving object using the multiple key points, and a correction unit that, when the correction parameters have been determined for the moving object for which the multiple key points have been detected from the captured images, corrects the multiple key points using the correction parameters.

[0006] The key point correction method of the present disclosure is executed by a computer. The control method includes an acquisition step of acquiring a plurality of captured images, a detection step of detecting a plurality of key points of a moving object from each of the captured images, a determination step of determining whether or not the plurality of key points of the moving object detected from the captured images satisfy a predetermined appropriateness condition, a determination step of determining correction parameters to be used for correcting the plurality of key points detected for the moving object using the plurality of key points when it is determined that the plurality of key points of the moving object satisfy the appropriateness condition, and a correction step of correcting the plurality of key points using the correction parameters when the correction parameters have been determined for the moving object from which the plurality of key points have been detected in the captured images.

[0007] The program of the present disclosure causes a computer to execute the keypoint correction method of the present disclosure. Effect of the Invention

[0008] According to the present invention, there is provided a technique for improving the accuracy of key points of a moving object detected from a captured image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of an outline of the operation of the keypoint correction device of the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of key points of a target object. [Diagram 3] 1 is a block diagram illustrating a functional configuration of a keypoint correction device according to a first embodiment. [Figure 4] FIG. 2 is a block diagram illustrating a hardware configuration of a computer that realizes the keypoint correction device. [Diagram 5] 4 is a flowchart illustrating a flow of processing executed by the keypoint correction device of the first embodiment. [Figure 6] 1 is a diagram illustrating a method for determining a rectangular region representing a target object. [Figure 7] 11 is a flowchart showing an example of the process flow of a key point correction device having a function of updating correction parameters. [Figure 8] 11A and 11B are diagrams illustrating a method for correcting keypoint corresponding points using a correction rectangle; [Figure 9] 1 is a block diagram illustrating a functional configuration of a key point correction device having a determination range generation unit. [Figure 10] 11 is a flowchart illustrating a process for generating a decision range using a teacher image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are given the same reference numerals, and duplicated explanations are omitted as necessary for clarity of explanation. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage unit accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or more arbitrary number of storage devices.

[0011] [Embodiment 1] <Summary> Fig. 1 is a diagram illustrating an example of an outline of the operation of the keypoint correction device 2000 of the first embodiment. Here, Fig. 1 is a diagram for easily understanding the outline of the keypoint correction device 2000, and the operation of the keypoint correction device 2000 is not limited to that shown in Fig. 1.

[0012] The keypoint correction device 2000 detects keypoints of a target object 20 from a plurality of captured images 10 in time series (for example, a plurality of video frames constituting video data). The target object 20 is a predetermined type of moving object that is treated as a target for detecting keypoints. The predetermined type is, for example, a vehicle, a person, or a mobile robot. Various types of vehicles can be handled as vehicles, such as any automobile such as a bus, a truck, or a passenger car, a motorcycle, or a bicycle. Note that the type of moving object that the keypoint correction device 2000 handles as the target object 20 is not limited to one type, and multiple types of moving objects may be handled as the target object 20. For example, the keypoint correction device 2000 detects any type of vehicle and a person as the target object 20.

[0013] The captured image 10 is generated by a camera 40 that is provided so as to capture an image of a target object 20. For example, when a vehicle or a person is treated as the target object 20, the camera 40 is provided at a location where it can capture an image of a road. The camera 40 may be a video camera that generates video data, or a still camera that generates still image data. In the latter case, the camera 40 repeatedly captures images to generate a plurality of captured images 10 in time series.

[0014] The key points of the target object 20 are feature points of the target object 20 detected from the captured image 10, and can be used to represent the position and orientation of the target object 20. For example, the key points are points obtained by projecting the vertices of the bottom surface of a three-dimensional bounding box that encloses the target object 20 onto the road.

[0015] Fig. 2 is a diagram illustrating key points of a target object 20. Fig. 2 shows a captured image 10 and an orthoimage 50 obtained by orthogonal transformation of the captured image 10. The orthogonal transformation for obtaining the orthoimage 50 from the captured image 10 can be realized by using a predetermined homography matrix.

[0016] The bounding box 60 is a three-dimensional bounding box that encloses the target object 20 detected from the captured image 10. Key point 30-1, key point 30-2, key point 30-3, and key point 30-4 are key points of the target object 20, and are obtained by projecting each vertex of the bottom surface of the bounding box 60 onto the road.

[0017] The keypoint correspondences 80 represent the keypoints 30 projected onto the orthoimage 50 by orthogonal transformation. The keypoint correspondences 80-1, 80-2, 80-3, and 80-4 correspond to the keypoints 30-1, 30-2, 30-3, and 30-4, respectively.

[0018] The key points 30 are not limited to the points obtained by projecting each vertex of the bottom surface of the three-dimensional bounding box of the target object 20 onto the road. For example, each vertex of the three-dimensional bounding box of the target object 20 may be treated as a key point 30.

[0019] The key point correction device 2000 detects key points 30 of a target object 20 included in each captured image 10. When multiple target objects 20 are detected from one captured image 10, key points 30 are detected for each of the multiple target objects 20.

[0020] Furthermore, the key point correction device 2000 judges for each captured image 10 whether or not there is a target object 20 detected from the captured image 10 whose key point 30 satisfies a predetermined suitability condition. The suitability condition is a condition that is satisfied when there is a high probability that the key point 30 of the target object 20 has been accurately detected from the captured image 10 and the key point 30 is suitable for determining a correction parameter, which will be described later, for the target object 20. Specific contents of the suitability condition will be described later.

[0021] Here, when the key points 30 for a certain target object 20 detected from a certain captured image 10 satisfy the suitability condition, the captured image 10 is called a proper image corresponding to the target object 20. In other words, a proper image corresponding to a certain target object 20 is a captured image 10 in which the key points 30 detected from the target object 20 satisfy the suitability condition.

[0022] When it is determined that the key points 30 of the target object 20 detected from the captured image 10 satisfy the proper condition, the key point correction device 2000 determines the correction parameters to be used for correcting the key points 30 of the target object 20 corresponding to the captured image 10 (proper image). The correction parameters represent the relative positional relationship that should be satisfied by the multiple key points 30 detected for the corresponding target object 20. For example, the key point correction device 2000 uses the key point corresponding points 80 on the ortho-image 50 obtained from the proper image to determine a rectangular area on the ortho-image 50 that represents the target object 20 corresponding to the proper image. Then, the key point correction device 2000 determines the combination of the width and height of this rectangular area as the correction parameters corresponding to the target object 20. The correction parameters determine the relative positional relationship that should be satisfied by the key point corresponding points 80 for the target object 20. Here, the key point corresponding points 80 are points that correspond to the key points 30. Therefore, by determining the relative positional relationship that the keypoint corresponding points 80 of the target object 20 should satisfy, the relative positional relationship that the keypoints 30 of the target object 20 should satisfy is also determined.

[0023] When correction parameters have been determined for a target object 20 detected from a captured image 10, the key point correction device 2000 corrects the key points 30 of the target object 20 detected from the captured image 10 using the correction parameters determined for the target object 20. In other words, when a target object 20 is detected from a captured image 10 that was generated after a correct image for the target object 20, the key point correction device 2000 corrects the key points 30 of the target object 20 detected from the captured image 10 using the correction parameters determined for the target object 20.

[0024] <Examples of effects> The key point correction device 2000 detects key points 30 of the target object 20 from the captured image 10. Here, the accuracy of the process of detecting the key points 30 from the captured image 10 may vary depending on how the target object 20 appears in the captured image 10. For example, in a situation where the position and orientation of the target object 20 are such that only one side of the target object 20 can be imaged by the camera 40, the target object 20 can only be captured two-dimensionally by the camera 40, and the accuracy of the process of detecting the key points 30 of the target object 20 may be low. On the other hand, in a situation where the position and orientation of the target object 20 are such that the top surface and two side surfaces of the target object 20 can be imaged by the camera 40, the target object 20 can be captured three-dimensionally by the camera 40, and the accuracy of the process of detecting the key points 30 of the target object 20 may be high.

[0025] In view of such a situation in which differences in the accuracy of the key points 30 may occur, the key point correction device 2000 uses the key points 30 that are considered to have been detected with high accuracy to correct the other key points 30. Here, appropriateness conditions are determined in advance so that the appropriateness conditions are satisfied when the key points 30 are considered to have been detected with high accuracy. Then, when the key points 30 of the target object 20 detected from the captured image 10 satisfy the appropriateness conditions, the key point correction device 2000 uses the key points 30 to determine the correction parameters of the target object 20. In this way, the correction parameters are determined by the key points 30 that are considered to have been detected with high accuracy. Furthermore, when the target object 20 for which the correction parameters have been determined is detected from the captured image 10, the key point correction device 2000 corrects the key points 30 of the target object 20 using the correction parameters. In this way, the other key points 30 are corrected based on the key points 30 that are considered to have been detected with high accuracy. Therefore, the accuracy of the correction parameters can be improved by using the key point correction device 2000.

[0026] The keypoint correction device 2000 of this embodiment will be described in more detail below.

[0027] <Example of functional configuration> FIG. 3 is a block diagram illustrating a functional configuration of the key point correction device 2000 of the first embodiment. The acquisition unit 2020 acquires a plurality of captured images 10. The detection unit 2040 detects key points 30 of the target object 20 included in each captured image 10. The determination unit 2060 determines whether or not the key points 30 of the target object 20 detected from the captured image 10 satisfy the appropriateness condition for use in determining the correction parameters. If the key points 30 of the target object 20 detected from the captured image 10 satisfy the appropriateness condition, the determination unit 2080 determines the correction parameters for the target object 20 using the key points 30. If the correction parameters corresponding to the target object 20 detected from the captured image 10 have been determined, the correction unit 2100 corrects the key points 30 of the target object 20 detected from the captured image 10 using the correction parameters corresponding to the target object 20.

[0028] <Example of hardware configuration> Each functional component of the keypoint correction device 2000 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a further explanation will be given of the case where each functional component of the keypoint correction device 2000 is realized by a combination of hardware and software.

[0029] FIG. 4 is a block diagram illustrating a hardware configuration of a computer 500 that realizes the keypoint correction device 2000. The computer 500 is any computer. For example, the computer 500 is a stationary computer such as a PC (Personal Computer) or a server machine. In addition, for example, the computer 500 is a portable computer such as a smartphone or a tablet terminal. The computer 500 may be provided inside the camera 40 that generates the captured image 10. The computer 500 may be a dedicated computer designed to realize the keypoint correction device 2000, or may be a general-purpose computer.

[0030] For example, by installing a specific application on the computer 500, each function of the keypoint correction device 2000 is realized on the computer 500. The application is configured with a program for realizing each functional component of the keypoint correction device 2000. The method of acquiring the program is arbitrary. For example, the program can be acquired from a storage medium (such as a DVD disk or a USB memory) in which the program is stored. Alternatively, the program can be acquired by downloading the program from a server device that manages the storage device in which the program is stored.

[0031] The computer 500 has a bus 502, a processor 504, a memory 506, a storage device 508, an input / output interface 510, and a network interface 512. The bus 502 is a data transmission path for the processor 504, the memory 506, the storage device 508, the input / output interface 510, and the network interface 512 to transmit and receive data to and from each other. However, the method of connecting the processor 504 and other components to each other is not limited to a bus connection.

[0032] The processor 504 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 506 is a main storage device realized using a random access memory (RAM) or the like. The storage device 508 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read only memory (ROM) or the like.

[0033] The input / output interface 510 is an interface for connecting the computer 500 to an input / output device. For example, the input / output interface 510 is connected to an input device such as a keyboard and an output device such as a display device.

[0034] The network interface 512 is an interface for connecting the computer 500 to a network 600. The network 600 may be a LAN (Local Area Network) or a WAN (Wide Area Network). For example, in FIG. 4, the computer 500 is communicatively connected to the camera 40 via the network 600. However, the method of communicatively connecting the computer 500 and the camera 40 is not limited to a connection via a network. In addition, the computer 500 may not be communicatively connected to the camera 40 as long as it can acquire the captured image 10 generated by the camera 40.

[0035] The storage device 508 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the keypoint correction device 2000. The processor 504 reads this program into the memory 506 and executes it to realize each functional component of the keypoint correction device 2000.

[0036] The keypoint correction device 2000 may be realized by one computer 500 or by multiple computers 500. In the latter case, the configurations of the computers 500 do not need to be the same, and may be different from each other.

[0037] <Processing flow> Fig. 5 is a flowchart illustrating a process performed by the keypoint correction device 2000 of the first embodiment. Note that the process shown in Fig. 5 is merely an example, and the process performed by the keypoint correction device 2000 is not limited to the process shown in Fig. 5.

[0038] Steps S102 to S120 constitute a loop process L1. The loop process L1 is repeatedly executed until a predetermined end condition is satisfied. The end condition of the loop process L1 is arbitrary. For example, the end condition may be that "a new captured image 10 does not exist" or that "a predetermined user operation is performed."

[0039] The acquisition unit 2020 acquires a captured image i (S104). Here, the acquisition unit 2020 acquires the captured images 10 in ascending order of generation time point. The detection unit 2040 detects key points 30 of each of one or more target objects 20 from the captured image i (S106).

[0040] S108 to S118 constitute a loop process L2. The loop process L2 is executed for each target object 20 detected from the captured image i. In S108, the key point correction device 2000 determines whether or not there is any target object 20 detected from the captured image i that has not yet been subjected to the loop process L2. If the loop process L2 has already been executed for all the target objects 20, the process of FIG. 5 proceeds to S120. On the other hand, if there is any target object 20 that has not yet been subjected to the loop process L2, one of them is selected, and the loop process L2 is executed for the selected target object 20. The target object 20 selected here is denoted as a target object j.

[0041] The determination unit 2060 determines whether or not the correction parameters have already been determined for the key points 30 of the target object j (S110). If the correction parameters for the target object j have not yet been determined (S110: NO), the determination unit 2060 determines whether or not the key points 30 of the target object j satisfy the appropriateness condition (S112). If the key points 30 of the target object j satisfy the appropriateness condition (S112: YES), the determination unit 2080 determines the correction parameters for the key points 30 of the target object j based on the key points 30 of the target object j detected from the captured image i (S114). If the key points 30 of the target object j do not satisfy the appropriateness condition (S112: NO), the process of FIG. 5 proceeds to S118.

[0042] In S110, if the correction parameters for the target object j have already been determined (S110: YES), the correction unit 2100 corrects the key points 30 of the target object j detected from the captured image i based on the correction parameters (S116).

[0043] S118 is the end of the loop process L2, so the process in Fig. 5 proceeds to the beginning (S108) of the loop process L2.

[0044] When the loop process L2 has been executed for all the target objects 20 detected from the captured image i, the process in Fig. 5 proceeds to S120. Since S120 is the end of the loop process L1, the process in Fig. 5 proceeds to the beginning (S102) of the loop process L1.

[0045] The key point correction device 2000 may handle one or more cameras 40. In the latter case, the key point correction device 2000 performs the process shown in FIG. 5 on the multiple captured images 10 in time series generated by each of the multiple cameras 40.

[0046] <Acquisition of captured image 10: S104> The acquisition unit 2020 acquires the captured image 10 (S104). The method of acquiring the captured image 10 is arbitrary. For example, the captured image 10 is stored in advance in a storage unit in a form accessible from the key point correction device 2000. In this case, the acquisition unit 2020 acquires the captured image 10 by accessing the storage unit. Alternatively, for example, the acquisition unit 2020 may acquire the captured image 10 by receiving the captured image 10 transmitted by another device (e.g., the camera 40).

[0047] The acquisition unit 2020 may acquire the captured images 10 one by one, or may collectively acquire a plurality of captured images 10. In the latter case, for example, the acquisition unit 2020 periodically accesses a storage unit in which the captured images 10 are stored, and collectively acquires the captured images 10 that have not yet been acquired.

[0048] <Detection of key point 30: S106> The detection unit 2040 detects the key points 30 of the target object 20 from the captured image 10 (S106). To this end, the detection unit 2040 first detects the target object 20 from the captured image 10. Then, the detection unit 2040 detects the key points 30 for each target object 20 detected from the captured image 10. Here, various existing technologies can be used for the technology to detect a predetermined type of object from an image and the technology to detect key points for an object detected from an image. For example, a machine learning model (hereinafter, object detection model) trained to output key points of each target object 20 included in an image in response to an input of the image is prepared. The detection unit 2040 can obtain the key points 30 of each target object 20 included in the captured image 10 by inputting the captured image 10 to this object detection model.

[0049] Here, in order to be able to correct the key points 30 of the same object 20 detected from other captured images 10 using the correction parameters of the object 20 determined using the appropriate image, it is necessary to track the same object 20 across multiple captured images 10. For example, the detection unit 2040 determines whether or not the same object 20 has already been detected from other captured images 10 for each object 20 detected from the captured image 10, and assigns an identifier to the object 20 based on the determination result. For the object 20 determined to have already been detected from other captured images 10, the assigned identifier is used. On the other hand, for the object 20 not detected from other captured images 10, a new identifier is assigned. In this way, the same identifier is assigned to the same object 20 detected from each of the multiple captured images 10. Therefore, the same object 20 can be tracked across multiple captured images 10.

[0050] Here, various existing technologies can be used to determine whether the target objects 20 detected from the different captured images 10 are the same or not. For example, by comparing the feature amounts of the target objects 20 obtained from the different captured images 10, it can be determined whether they are the same or not. When using this method, for example, the object detection model used to detect the above-mentioned key points 30 is provided with a function to output the feature amounts of the detected target objects 20. This allows the detection unit 2040 to obtain the feature amounts and key points 30 for each target object 20 detected from the captured image 10.

[0051] <Determining whether the appropriate conditions are met: S112> The determination unit 2060 determines whether the keypoints 30 detected for the target object j satisfy the appropriateness condition (S112). The appropriateness condition is a condition that is satisfied when the keypoints 30 detected for the target object j are suitable for determining the correction parameters. Some specific examples of conditions that can be adopted as the appropriateness condition will be described below.

[0052] <<Example 1: Conditions regarding the position and orientation of the target object 20>> For example, as the appropriateness condition, a condition that "the position, size, orientation, or a combination of two or more of these of the target object j is included in a predetermined range" is used. Hereinafter, this predetermined range is referred to as a determination range. When the above-mentioned appropriateness condition is used, a range suitable for determining correction parameters for the position, size, orientation, or a combination of two or more of these of the target object j is preliminarily determined as a determination range. Then, when the position, size, orientation, or a combination of two or more of these of the target object j identified using the key points 30 detected from the captured image i is included in a range suitable for determining correction parameters (i.e., the determination range), it is determined that the appropriateness condition is satisfied. On the other hand, when the position, size, orientation, or a combination of two or more of these of the target object j identified using the key points 30 detected from the captured image i is not included in the determination range, it is determined that the appropriateness condition is not satisfied.

[0053] When the range of the position of the target object j is included in the determination range, the determination unit 2060 identifies the position of the target object j by using the key points 30 of the target object j. The position of the target object j can be represented, for example, by the position of a specific part (such as the center of gravity) of the target object j. For example, the position of the target object j is represented by the position of the center of gravity of the target object j. In this case, the determination unit 2060 calculates the position of the center of gravity of the multiple key points 30 of the target object j detected from the captured image i, and uses the position of the center of gravity as a parameter representing the position of the target object j.

[0054] Here, the position of the target object j is represented by coordinates in k-dimensional space (k is 2 or 3). In this case, the determination range for the position of the target object j is represented by a subspace in the k-dimensional space. In other words, the appropriateness condition that "the position of the target object j is included in the determination range" is satisfied when the coordinates in the k-dimensional space that represent the position of the target object j detected from the captured image i are included in the determination range, which is a subspace in the k-dimensional space.

[0055] When the range of the size of the target object j is included in the determination range, the determination unit 2060 identifies the size of the target object j by using the key points 30 of the target object j detected from the captured image i. Here, there are various methods for expressing the size of the target object j. For example, the key points 30 represent the vertices of the bottom surface of the three-dimensional bounding box of the target object j projected onto the road. In this case, the determination unit 2060 uses the key point corresponding points 80 on the orthoimage 50 to determine a rectangular area representing the target object j on the orthoimage 50, and uses the combination (w,h) of the width w and height h of the rectangular area as a parameter representing the size of the target object j.

[0056] There are various methods for determining a rectangular region representing a target object j on an orthoimage 50. A specific example will be described here with reference to Fig. 6. Fig. 6 is a diagram illustrating a method for determining a rectangular region representing a target object j. In Fig. 6, keypoint corresponding points 80 of the target object j are keypoint corresponding points 80-1 to 80-4.

[0057] First, the determination unit 2060 identifies the keypoint corresponding point 80 closest to the camera 40 among the keypoint corresponding points 80 of the target object j. In FIG. 6, the keypoint corresponding point 80 closest to the camera 40 is assumed to be the keypoint corresponding point 80-2. Next, the determination unit 2060 identifies the keypoint corresponding point 80 whose line segment connecting with the keypoint corresponding point 80-2 is longer (whose distance from the keypoint corresponding point 80-2 is longer) among the two keypoint corresponding points 80 adjacent to the keypoint corresponding point 80-2 in the quadrangle formed by the keypoint corresponding points 80-1 to 80-4. In FIG. 6, the keypoint corresponding points 80 adjacent to the keypoint corresponding point 80-2 are the keypoint corresponding point 80-1 and the keypoint corresponding point 80-3. The line segment connecting the keypoint corresponding point 80-1 and the keypoint corresponding point 80-2 is longer than the line segment connecting the keypoint corresponding point 80-3 and the keypoint corresponding point 80-2. Therefore, the identified keypoint corresponding point 80 is the keypoint corresponding point 80-1.

[0058] The determination unit 2060 determines the line segment connecting the identified keypoint corresponding point 80-1 and keypoint corresponding point 80-2 as the long side 90-1, and determines a rectangular area whose other long side 90-2 passes near the keypoint corresponding point 80-3 and keypoint corresponding point 80-4 as a rectangular area 100 representing the target object j. There are various methods for determining the long side 90-2. For example, the long side 90-2 is determined so that the distance between the long side 90-1 and the long side 90-2 (i.e., the length of the short side of the rectangular area 100) is the distance between the long side 90-1 and the keypoint corresponding point 80-3, the distance between the long side 90-1 and the keypoint corresponding point 80-4, or the average of these two. In the example of FIG. 6, the distance between the long side 90-1 and the long side 90-2 is the average of the distance between the long side 90-1 and the keypoint corresponding point 80-3 and the distance between the long side 90-1 and the keypoint corresponding point 80-4.

[0059] In this way, when the size of the target object j is expressed by width and height, the determination range for the size of the target object j is expressed as a subspace in two-dimensional space. In other words, when the size of the target object j is expressed as a combination of width and height (w,h), the determination range for the size of the target object j is expressed as a subspace in wh space. Therefore, the appropriateness condition that "the size of the target object j is included in the determination range" is satisfied when the size (w[i,j]),h[i,j]) of the target object j detected from the captured image i is included in the determination range, which is a subspace in wh space.

[0060] The size of the target object j may be expressed in three dimensions. For example, assume that eight vertices of the three-dimensional bounding box of the target object j are detected as key points 30. In this case, the size of the target object j can be expressed as a combination (w, d, h) of the width w, depth d, and height h of the three-dimensional bounding box. In this case, the determination range for the size of the target object j is expressed as a subspace in three-dimensional space. Therefore, the appropriateness condition that "the size of the target object j is included in the determination range" is satisfied when the size (w[i, j], d[i, j], h[i, j]) of the target object j detected from the captured image i is included in the determination range, which is a subspace in the hdw space.

[0061] When the range of the posture of the target object j is included in the appropriate condition, the determination unit 2060 identifies the posture of the target object j by using the key points 30 of the target object j detected from the captured image i. Here, there are various methods for expressing the posture of the target object j. For example, the posture of the target object j is expressed by the posture angle θ of the target object j on the ortho-image 50. The posture angle of the target object j on the ortho-image 50 can be expressed, for example, by the angle that the reference axis of the target object j forms with respect to the reference axis (for example, the horizontal axis) of the ortho-image 50. For example, the long side or short side of the rectangular region 100 representing the target object j described in FIG. 6 can be used as the reference axis of the target object j.

[0062] When expressing the posture of target object j by its posture angle in this way, the determination range for the posture of target object j is expressed as a one-dimensional subspace (i.e., the numerical range of posture angle θ). Therefore, the appropriateness condition that "the posture of target object j is included in the determination range" is satisfied when the posture angle θ[i,j] of target object j detected from captured image i is included in the determination range, which is a predetermined numerical range.

[0063] The orientation of the target object j may be represented by an orientation in a three-dimensional space. For example, in this case, the orientation of the target object j can be represented by a combination of angles (pitch θ, roll φ, yaw ψ) that the three-dimensional bounding box of the target object j makes with respect to the reference axes in each of the three directions. In this case, the determination range for the orientation of the target object j is represented by a subspace in the three-dimensional space. Therefore, the appropriateness condition that "the orientation of the target object j is included in the determination range" is satisfied when the orientation (θ[i,j],φ[i,j],ψ[i,j]) of the target object j detected from the captured image i is included in the determination range, which is a subspace in the θφψ space.

[0064] As described above, the determination range may be determined for any combination of two or more of the position, size, and orientation of the target object j. For example, the determination range is determined as the range of the combination (x, y, θ) of the position (x, y) and orientation θ of the target object j. In this case, the determination unit 2060 determines whether or not the combination (x[i, j], y[i, j], θ[i, j]) of the position and size of the target object j detected from the captured image i is included in the determination range.

[0065] Alternatively, for example, the determination range is defined as a range of a combination (x, y, w, h, θ) of the position (x, y), size (w, h), and orientation θ of a target object j. In this case, the determination unit 2060 determines whether or not the combination (x[i, j], y[i, j], w[i, j], h[i, j], θ[i, j]) of the position, size, and orientation of the target object j detected from the captured image i is included in the determination range.

[0066] In addition, when the determination range includes a range related to the size of the target object j, if the position and orientation of the target object j suitable for determining the correction parameters differ for target objects j of different sizes, such as a bus, a truck, a passenger car, a person, etc., a different determination range can be set for each type of target object j. On the other hand, if there is no significant difference in the range of positions and orientations suitable for determining the correction parameters even if the size of the target object j differs, the amount and time of calculations required to determine whether the appropriateness condition is satisfied can be reduced by not including the size of the target object j in the determination range.

[0067] The method for determining the decision range will be described later.

[0068] <<Example 2: Conditions for time stability of key point 30>> As another example of the appropriateness condition, a condition that "the time stability of the key points 30 of the target object j is high" is used. Here, the high time stability of the key points 30 means that the change in time of the key points 30 is small. When this appropriateness condition is used, the determination unit 2060 calculates an index value representing the time stability (hereinafter, a time stability index value) for the key points 30 of the target object j detected from the captured image i. In this case, a more specific appropriateness condition can be used, which is a condition that "the time stability index value calculated for the key points 30 of the target object j is equal to or less than a threshold value." Therefore, when the time stability index value is equal to or less than the threshold value, it is determined that the detection condition is satisfied. On the other hand, when the time stability index value is greater than the threshold value, it is determined that the detection condition is not satisfied.

[0069] The time stability index value is calculated based on, for example, a change in size of a target object j in a captured image i over time. For example, the time stability evaluation value is calculated using the following formula (1).

number

[0070] Furthermore, when the captured images 10 are generated at regular time intervals (when the camera 40 captures images at regular intervals), the time stability evaluation value can be calculated by the following formula (2).

number

[0071] Here, when calculating the time stability index value, the magnitude of the reliability of the key point 30 may be taken into consideration. In this case, the time stability index value can be expressed by, for example, the following formula (3).

number

[0072] Here, various existing methods can be used to calculate the reliability of the keypoint 30. For example, if keypoints are tracked using filtering such as a Kalman filter, the reliability of the keypoints can be calculated from the covariance matrix of the state variables.

[0073] <<Example 3: Combination of Example 1 and Example 2>> The determination unit 2060 may use a combination of the above-mentioned examples 1 and 2. Hereinafter, the suitability condition described in example 1 will be referred to as suitability condition 1, and the suitability condition described in example 2 will be referred to as suitability condition 2. For example, when both suitability condition 1 and suitability condition 2 are satisfied, the determination unit 2060 determines that the suitability condition is satisfied. On the other hand, when at least one of suitability condition 1 and suitability condition 2 is not satisfied, the determination unit 2060 determines that the suitability condition is satisfied.

[0074] For example, first, the determination unit 2060 calculates the position of the object j detected from the captured image i, and determines whether the calculated position is included in the determination range (i.e., whether the appropriateness condition 1 is satisfied). If the appropriateness condition 1 is not satisfied, the determination unit 2060 determines that the appropriateness condition is not satisfied for the object j detected from the captured image i (S112: NO). On the other hand, if the appropriateness condition 1 is satisfied, the determination unit 2060 determines whether the time stability of the object j detected from the captured image i is high (i.e., whether the appropriateness condition 2 is satisfied). If the appropriateness condition 2 is not satisfied, the determination unit 2060 determines that the appropriateness condition is not satisfied for the object j detected from the captured image i (S112: NO). On the other hand, if the appropriateness condition 2 is satisfied, the determination unit 2060 determines that the appropriateness condition is satisfied for the object j detected from the captured image i (S112: YES).

[0075] Alternatively, for example, the determination unit 2060 may determine that the appropriateness conditions are satisfied when at least one of the appropriateness conditions 1 and 2 is satisfied. In this case, the appropriateness conditions are determined to be not satisfied only when both the appropriateness conditions 1 and 2 are not satisfied.

[0076] <Determining correction parameters: S114> The determination unit 2080 determines the correction parameters for the target object j using the key points 30 of the target object j detected from the captured image i corresponding to the target object j (S114). For example, the correction parameters are parameters representing the size of the target object j. For example, when the size of the target object j is represented by a width w and a height h, the determination unit 2080 determines the size (w[i,j],h[i,j]) of the target object j detected from the captured image i as the correction parameters.

[0077] The size of a target object j is represented by a width w, a depth d, and a height h. In this case, the determination unit 2080 determines the size (w[i,j],d[i,j],h[i,j]) of the target object j detected from the captured image i as the correction parameters.

[0078] The determination unit 2080 stores the determined correction parameters in a storage unit accessible from the keypoint correction device 2000 in association with the identification information of the corresponding target object j. The determination as to whether or not the correction parameters for the target object j have already been determined (S110) can be performed, for example, by determining whether or not the correction parameters corresponding to the target object j are stored in the storage unit.

[0079] <<Updating compensation parameters>> 5, for a target object j for which correction parameters have already been determined, a determination as to whether the appropriateness condition is satisfied is not performed. However, even if correction parameters have already been determined for a target object j detected from a captured image i, there is a possibility that a key point 30 of the target object j detected from the captured image i satisfies the appropriateness condition and a correction parameter more suitable than the current correction parameter can be obtained from the key point 30. Therefore, for example, the key point correction device 2000 may update the correction parameter when the correction parameter has already been determined.

[0080] Fig. 7 is a flowchart showing an example of the process flow of the keypoint correction device 2000 having a function of updating the correction parameters. In Fig. 7, when the correction parameters of the target object j have already been determined (S110), it is determined whether or not to update the correction parameters (S202). Then, when it is determined that the correction parameters are to be updated (S202: YES), new correction parameters are determined (S114).

[0081] In S202, there are various methods for determining whether or not to update the correction parameters. For example, in the case where the above-mentioned proper condition 1 is used, it is assumed that a plurality of decision ranges with different priorities are defined. In this case, when the key point 30 of the target object j detected from the captured image i satisfies the proper condition and the correction parameters are determined using the key point 30, the determination unit 2080 sets the priority of the decision range corresponding to the proper condition as the priority of the correction parameters. Information indicating the priority of the correction parameters is stored in the storage unit together with the correction parameters.

[0082] For another captured image i obtained after that, in S202, the determination unit 2080 determines whether or not the key points 30 of the target object j detected from the captured image i satisfy the appropriateness condition. If the appropriateness condition is not satisfied, the determination unit 2080 determines not to update the correction parameters. On the other hand, if the appropriateness condition is satisfied, the determination unit 2080 further determines whether or not the priority of the determination range corresponding to the satisfied appropriateness condition is higher than the priority of the current correction parameters corresponding to the target object j. If the priority of the determination range corresponding to the satisfied appropriateness condition is higher than the priority of the current correction parameters corresponding to the target object j, the correction parameters are updated. On the other hand, if the priority of the determination range corresponding to the satisfied appropriateness condition is not higher than the priority of the current correction parameters corresponding to the target object j, the correction parameters are not updated.

[0083] For example, it is assumed that a determination range R1 for priority 1, a determination range R2 for priority 2, and a determination range R3 for priority 3 are determined. Here, the higher the priority, the higher the priority. In this case, it is assumed that the position of the target object j detected from the captured image i is included in the determination range R2, so that the appropriateness condition is satisfied. It is also assumed that the correction parameters for the target object j have not yet been determined. In this case, the determination unit 2080 determines the correction parameters using the key points 30 of the target object j detected from the captured image. In this case, the priority of the correction parameters is 2.

[0084] After that, the position of the target object j detected from another captured image i is included in the decision range R1, so the appropriateness condition is satisfied. In this case, the priority of the appropriateness condition satisfied here is 1, which is lower than the priority of the current correction parameters corresponding to the target object j, which is 2. Therefore, the correction parameters are not updated.

[0085] After that, it is assumed that the position etc. of object j detected from another captured image i is included in the determination range R3, so that the appropriateness condition is satisfied. In this case, the priority of the satisfied appropriateness condition is 3, which is higher than the priority of 2 of the current correction parameters corresponding to object j. Therefore, the determination unit 2080 determines the correction parameters using the key point 30 of object j detected from this captured image i, and updates the correction parameters. A priority of 3 is associated with this correction parameter.

[0086] In the case where the above-mentioned appropriateness condition 2 is used, it is possible to determine whether or not to update the correction parameters based on the magnitude of the temporal stability index value. Specifically, when the key point 30 of the target object j detected from the captured image i satisfies the appropriateness condition 2 and the correction parameters are determined using the key point 30, the determination unit 2080 sets the calculated temporal stability index value as a value representing the priority of the correction parameters. Here, since the smaller the temporal stability index value, the more preferable the value, the smaller the temporal stability index value, the higher the priority.

[0087] Thereafter, when the key point 30 of the target object j detected from another captured image i satisfies the appropriateness condition 2, the determination unit 2080 judges whether the calculated time stability index value is smaller than the value set as the priority of the current correction parameter corresponding to the target object j. When the calculated time stability index value is smaller than the priority value, the correction parameter is updated. On the other hand, when the calculated time stability index value is not smaller than the priority value, the correction parameter is not updated.

[0088] However, in the case where the time stability index value is slightly smaller than the priority value, although a more appropriate correction parameter is obtained, the effect of updating the correction parameter may not be large. Therefore, for example, the correction parameter may be updated only when the calculated time stability index value is smaller than the priority value and the difference therebetween is sufficiently large (above a threshold value).

[0089] <Correction of key point 30: S116> When a target object j is detected from a captured image i generated after the captured image 10 corresponding to the target object j (S110: YES), the correction unit 2100 corrects the key points 30 of the target object j detected from the captured image i using the correction parameters corresponding to the target object j (S116). Here, the captured image 10 to which the key points 30 are to be corrected is called a correction target image.

[0090] Assume that the size (w, h) of the target object j is determined as a correction parameter corresponding to the target object j. In this case, the correction unit 2100 orthogonally transforms the correction target image to calculate the key point corresponding points 80 on the ortho image 50 for each key point 30 of the target object j on the captured image i. Furthermore, the correction unit 2100 generates a correction rectangle, which is a rectangular area of ​​size (w, h), based on the arrangement of the key point corresponding points 80 on the ortho image 50, and corrects the key point corresponding points 80 so that the key point corresponding points 80 become each vertex of the correction rectangle. Thereafter, the correction unit 2100 performs an inverse transformation to return the ortho image 50 to the correction target image, thereby converting each corrected key point corresponding point 80 into a point on the correction target image. The points on the correction target image obtained in this manner are used as the corrected key points 30.

[0091] Here, a method of correcting the keypoint corresponding points 80 will be described. For example, the correction unit 2100 associates each keypoint corresponding point 80 of the target object 20 with one of the vertices of a correction rectangle of size (w, h). Furthermore, the correction unit 2100 determines the arrangement of the correction rectangle so that the sum of the distances between the corresponding points is minimized. Then, the correction unit 2100 changes the position of each keypoint corresponding point 80 to the position of the corresponding vertex of the correction rectangle. Furthermore, if there is no keypoint corresponding point 80 corresponding to a vertex of the correction rectangle, the correction unit 2100 generates a new keypoint corresponding point 80 corresponding to that vertex. In this way, the correction rectangle is used to correct the position of the keypoint corresponding point 80 and generate the keypoint corresponding point 80, thereby correcting the keypoint corresponding point 80.

[0092] Fig. 8 is a diagram illustrating a method of correcting keypoint corresponding points 80 using a correction rectangle. The case in the upper part of Fig. 8 is a case where four keypoint corresponding points 80 exist. In this case, the arrangement of the correction rectangle 110 is determined based on the positions of the keypoint corresponding points 80-1, 80-2, 80-3, and 80-4. Then, the positions of the keypoint corresponding points 80-1, 80-2, 80-3, and 80-4 are corrected to the positions of the vertices 120-1, 120-2, 120-3, and 120-4, respectively.

[0093] 8 is a case where there are three keypoint corresponding points 80. In this case, the arrangement of the corrected rectangle 110 is determined based on the positions of the keypoint corresponding point 80-1, the keypoint corresponding point 80-2, and the keypoint corresponding point 80-3. Then, the positions of the keypoint corresponding point 80-1, the keypoint corresponding point 80-2, and the keypoint corresponding point 80-3 are corrected to the positions of the vertex 120-1, the vertex 120-2, and the vertex 120-3, respectively. Furthermore, a new keypoint corresponding point 80 is generated at the position of the vertex 120-4.

[0094] 8 is a case where there are two keypoint corresponding points 80. In this case, the arrangement of the corrected rectangle 110 is determined based on the positions of the keypoint corresponding points 80-1 and 80-2. Then, the positions of the keypoint corresponding points 80-1 and 80-2 are corrected to the positions of the vertices 120-2 and 120-3, respectively. Furthermore, new keypoint corresponding points 80 are generated at the positions of the vertices 120-1 and 120-4.

[0095] Here, when there are two keypoint corresponding points 80 on the target object 20, the arrangement of the correction rectangle 110 differs depending on which part (front right, front left, rear right, or rear left) of the target object 20 each keypoint corresponding point 80 represents. For example, in the case of the lower part of Fig. 8, it is also possible to arrange the keypoint corresponding point 80-1 and the keypoint corresponding point 80-2 to correspond to the vertex 120-1 and the vertex 120-4, respectively.

[0096] Therefore, for example, the detection unit 2040 is configured to specify which part of the target object 20 each key point 30 represents when detecting the key points 30. The correction unit 2100 then obtains information indicating which part of the target object 20 each key point 30 represents, and associates the vertices 120 of the corrected rectangle 110 with the key point corresponding points 80 using the information.

[0097] Any method may be used to configure the detection unit 2040 so as to identify which part of the target object 20 each key point 30 represents. For example, as described above, it is assumed that the key points 30 are detected using an object detection model. In this case, the object detection model is configured to output the key points 30 in a manner that makes it possible to know which part of the target object 20 each key point 30 represents. As a more specific example, it is assumed that the object detection model is configured as a neural network. In this case, the meaning of each output node of the object detection model is determined when the object detection model is trained. Therefore, the correction unit 2100 can identify which part of the target object 20 each key point corresponding point 80 represents, depending on which output node of the object detection model the key point 30 corresponding to each key point corresponding point 80 of the target object 20 is output from.

[0098] <Result output> The keypoint correction device 2000 outputs information indicating the processing result. Hereinafter, the information output by the keypoint correction device 2000 is referred to as output information. The output information includes at least information representing the keypoints 30 of each target object 20 detected from each captured image 10. The information representing the keypoints 30 is, for example, the coordinates of the keypoints 30 on the target object 20.

[0099] Here, when the key points 30 are corrected by the correction unit 2100, the output information is configured to include at least information representing the corrected key points 30. In this case, the output information may or may not include information representing the pre-correction key points 30.

[0100] The output information may include an image generated by superimposing an image representing the key points 30 or a three-dimensional bounding box of the target object 20 on the captured image 10. Here, in the image, the uncorrected key points 30 and the corrected key points 30 may be distinguishable from each other. For example, the marks representing the corrected key points 30 and the marks representing the corrected key points 30 may be made distinguishable by making them different colors from each other.

[0101] <How to generate decision range> When using the above-mentioned appropriateness condition 1, it is necessary to generate a decision range in advance. A method for generating a decision range will be described below. Here, a device having a function for generating a decision range is called a decision range generating device. The decision range generating device may be realized integrally with the keypoint correction device 2000 (they are both realized by the same computer), or may be realized separately from the keypoint correction device 2000 (they are both realized by different computers).

[0102] When the decision range generating device is provided integrally with the keypoint correction device 2000, the keypoint correction device 2000 has a decision range generating unit that realizes the function of the decision range generating device. FIG. 9 is a block diagram illustrating a functional configuration of the keypoint correction device 2000 having the decision range generating unit. In the following, for the sake of simplicity, the subject of generating the decision range is the decision range generating unit 2120. When the decision range generating device is provided separately from the keypoint correction device 2000, the subject of each process described below is the decision range generating device.

[0103] For example, the decision range can be generated by clustering using a teacher image. The teacher image is a captured image obtained from a camera 40. Correct keypoints are added to the teacher image. The correct keypoints in the teacher image represent keypoints to be detected from the teacher image. For example, the correct keypoints are manually determined by a user of the decision range generation device.

[0104] FIG. 10 is a flowchart illustrating the flow of a process for generating a decision range using a teacher image. S302 to S316 constitute a loop process L3. The loop process L3 is executed for each teacher image. In S302, the decision range generating unit 2120 determines whether or not there is a teacher image that has not yet been the target of the loop process L3. If the loop process L3 has already been executed for all teacher images, the process in FIG. 10 proceeds to S318. On the other hand, if there is a teacher image that has not yet been the target of the loop process L3, one of them is selected, and the loop process L3 is executed for the selected teacher image. The teacher image selected here is denoted as teacher image k.

[0105] The determination range generation unit 2120 detects the key points 30 of each of the one or more target objects 20 from the teacher image k (S304). The process of detecting the key points 30 of each target object 20 from the teacher image k is realized by a process similar to the process of the detection unit 2040 detecting the key points 30 of each target object 20 from the captured image 10. Therefore, for example, the detection unit 2040 is used for the process of detecting key points from the teacher image k.

[0106] S306 to S314 constitute a loop process L4. The loop process L4 is executed for each target object 20 detected from the teacher image k. In S306, the determination range generation unit 2120 determines whether or not there is any target object 20 detected from the teacher image k that has not yet been the target of the loop process L4. If the loop process L4 has already been executed for all the target objects 20, the process of FIG. 10 proceeds to S316. On the other hand, if there is a target object 20 that has not yet been the target of the loop process L4, one of them is selected, and the loop process L4 is executed for the selected target object 20. The target object 20 selected here is denoted as a target object m.

[0107] The determination range generating unit 2120 calculates an error index value representing an error between the key point 30 detected for the target object m and the correct key point for the target object m (S308). For example, the error index value is a statistical value (such as an average value) of the magnitude of the distance between the corresponding key points 30 and the correct key point. For example, it is assumed that correct key points A1, A2, A3, and A4 are set for the target object m of the teacher image k. It is also assumed that B1, B2, B3, and B4 are detected as the key points 30 of the target object m from the teacher image k. The determination range generating unit 2120 first associates the correct key points with the key points 30. For example, the determination range generating unit 2120 associates each correct key point of the target object m with the key point 30 of the target object m that is closest to the correct key point. Here, it is assumed that keypoints B1, B2, B3, and B4 are associated with correct keypoints A1, A2, A3, and A4, respectively. In this case, the determination range generating unit 2120 calculates the statistical values ​​of the distance between A1 and B1, the distance between A2 and B2, the distance between A3 and B3, and the distance between A4 and B4 as error index values.

[0108] The decision range generating unit 2120 determines whether the error index value calculated for the target object m is equal to or less than a threshold value (S310). If the error index value is equal to or less than the threshold value (S310: YES), the decision range generating unit 2120 calculates a feature vector using key points 30 of the target object m detected from the teacher image k.

[0109] A feature vector represents features such as the position and orientation of a target object m. For example, the features of a target object m may be its position (x, y), size (w, h), and orientation θ. In this case, the feature vector is a five-dimensional vector (x, y, w, h, θ).

[0110] After the feature vector is calculated, or after it is determined that the error index value is greater than the threshold value (S310: NO), the process in Fig. 10 proceeds to S314. Since S314 is the end of loop process L4, the process in Fig. 10 proceeds to the beginning of loop process L4 (S306).

[0111] After the loop process L4 for the teacher image k is completed, the process in Fig. 10 proceeds to S316. Since S316 is the end of the loop process L3, the process in Fig. 10 proceeds to the beginning of the loop process L3 (S302).

[0112] After loop processing L3 is performed for each teacher image, the decision range generating unit 2120 performs clustering of the multiple feature vectors calculated so far (S318). Here, the error index value being equal to or less than the threshold means that the key points 30 of the target object m detected from the teacher image k are close to the correct key points (in other words, the key points 30 are detected with high accuracy). Therefore, when the error index value is equal to or less than the threshold, the feature vector of the target object m detected from the teacher image k represents the position, orientation, etc. of the target object m where accurate key points can be detected. Therefore, the decision range generating unit 2120 uses the feature vector obtained when the error index value is equal to or less than the threshold in this way to specify the range of the position and orientation of the target object where accurate key points can be detected, and treats the range as the decision range.

[0113] A specific method for classifying a plurality of vectors by clustering them can utilize various existing techniques. For example, the determination range generating unit 2120 classifies the feature vectors into a plurality of clusters by approximating the distribution of the feature vectors using a Gaussian mixture distribution expressed by the following formula (4).

number

[0114] Here, each centroid c_u that minimizes E can be determined using, for example, an EM (Expectation-Maximization) method such as the k-means method. In addition, various existing technologies can be used to determine an appropriate total number of clusters in clustering.

[0115] The determination range generating unit 2120 generates a determination range based on the result of the clustering (S320). Specifically, the determination range generating unit 2120 first calculates a stability index value representing the stability of each of the multiple clusters generated by the clustering. The stability index value can be calculated as a value that increases as the degree of dispersion of the feature vectors included in the cluster decreases. For example, the stability index value is calculated using the following formula (5).

number

[0116] The decision range generating unit 2120 generates a decision range from clusters whose stability index value is equal to or greater than a threshold value. If there are multiple clusters whose stability index value is equal to or greater than a threshold value, a decision range is generated for each cluster. When multiple decision ranges are generated in this manner, the above-mentioned priority may be assigned to each decision range. For example, a higher priority is assigned to a cluster having a higher stability index value.

[0117] Here, a method for generating a determination range from a cluster will be described. For example, the determination range generation unit 2120 calculates the centroid of the cluster and sets the vicinity area of the centroid as the determination range. As the vicinity area of the centroid, for example, for each element of the feature vector, an area given a predetermined range from the value of the centroid can be used. For example, assume that the feature vector is represented by a 5D vector (x, y, w, h, θ) representing the position (x, y), size (w, h), and orientation θ of the target object 20. Also, assume that the centroid of the cluster is (xc, yc, wc, hc, θc). For example, in this case, an area of "xc - α < x < xc + α, and yc - α < y < yc + α, and wc - α < w < wc + α, and hc - α < h < hc + α, and θc - α < θ < θc + α", which is an area with a predetermined width from the centroid for each element, can be used as the determination range. Note that in this example, the size of the range centered on the centroid is the same α for all elements, but the size of the range may be different for each element. For example, the range for each element may be determined using the degree of dispersion (variance or standard deviation) of each element in the cluster.

[0118] Also, the determination range generation unit 2120 may generate the determination range using only a part of the elements used for generating the cluster. For example, as described above, when the feature vector is represented by a 5D vector representing the position, size, and orientation of the target object 20, the determination range generation unit 2120 may generate the determination range as the range of the 3D space representing the position and orientation of the target object 20. This corresponds to generating the determination range from a subspace obtained by projecting the cluster in the 5D space onto the 3D space. For example, in the case of giving a predetermined width from the centroid to each element as described above, a determination range of "xc - α < x < xc + α, and yc - α < y < yc + α, and θc - α < θ < θc + α" can be generated.

[0119] As described above, the present invention has been described with reference to the embodiments, but the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0120] In the above example, the program includes a set of instructions (or software code) for making the computer perform one or more functions described in the embodiment when the program is loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagating signals.

[0121] (Appendix 1) An acquisition unit that acquires a plurality of captured images; A detection unit that detects a plurality of key points of a moving object from each of the captured images; a determination unit that determines whether or not the plurality of key points of the moving object detected from the captured image satisfy a predetermined appropriate condition; a determination unit that, when it is determined that the plurality of key points of the moving object satisfy the appropriateness condition, determines correction parameters to be used for correcting the plurality of key points detected for the moving object, using the plurality of key points; and a correction unit that, when the correction parameters have been determined for the moving object from which the multiple key points have been detected from the captured image, corrects the multiple key points using the correction parameters. (Appendix 2) the appropriateness condition indicates a determination range that is a range that the position, size, orientation, or a combination of two or more of the moving object should satisfy; The keypoint correction device described in Appendix 1, wherein the determination unit uses multiple keypoints of the moving object to identify a position, size, or orientation of the moving object, and determines that the appropriateness condition is satisfied when the identified position, size, orientation, or a combination of two or more of these is included in the determination range. (Appendix 3) The determination unit is orthogonally transforming the plurality of key points of the moving object detected from the captured image to calculate key point corresponding points on the orthoimage corresponding to each of the plurality of key points; A keypoint correction device as described in Appendix 2, which calculates the position, size, or orientation of the moving object on the orthoimage based on the calculated positions of the multiple keypoint corresponding points as the position, size, or orientation of the moving object. (Appendix 4) a decision range generating unit that generates the decision range indicated by the appropriate condition by using a plurality of teacher images in which the correct key points for the moving object are determined; The determination range generation unit For each of the plurality of teacher images, an error between the correct key point in the teacher image and the key point of the moving object detected from the teacher image is calculated, and if the error is equal to or smaller than a threshold, a feature vector of the key point of the moving object is calculated; A keypoint correction device as described in Appendix 3, which generates the decision range using the feature vectors included in a cluster whose stability is equal to or greater than a threshold, among a plurality of clusters obtained by clustering the calculated plurality of feature vectors. (Appendix 5) The keypoint correction device described in Appendix 1, wherein the determination unit calculates a time stability index value representing temporal stability for the multiple keypoints of the moving object, and determines that the appropriate condition is satisfied if the time stability index value is equal to or greater than a threshold. (Appendix 6) 6. The keypoint correction device according to claim 5, wherein the time stability index value is larger as the amount of time change in size of the moving object within a predetermined time period is smaller. (Appendix 7) 7. The keypoint correction device according to claim 6, wherein the time stability index value is larger as the reliability of the multiple keypoints of the moving object is higher. (Appendix 8) the correction parameter represents a size of the moving object; The keypoint correction device according to any one of appendixes 1 to 7, wherein, when the correction parameters have been determined for the moving object for which multiple keypoints have been detected from the captured image, the correction unit corrects one or more of the keypoints so that the size of the moving object represented by the multiple keypoints becomes the size represented by the correction parameters. (Appendix 9) 1. A computer implemented method for keypoint correction, comprising: An acquisition step of acquiring a plurality of captured images; a detection step of detecting a plurality of key points of a moving object from each of the captured images; a determination step of determining whether or not the plurality of key points of the moving object detected from the captured image satisfy a predetermined appropriate condition; a determining step of determining, when it is determined that the plurality of key points of the moving object satisfy the appropriateness condition, a correction parameter to be used for correcting the plurality of key points detected for the moving object, using the plurality of key points; and when the correction parameters have been determined for the moving object from which the multiple key points have been detected from the captured image, a correction step of correcting the multiple key points using the correction parameters. (Appendix 10) the appropriateness condition indicates a determination range that is a range that the position, size, orientation, or a combination of two or more of the moving object should satisfy; 10. The keypoint correction method of claim 9, wherein in the determination step, the position, size, or orientation of the moving object is identified using a plurality of the keypoints of the moving object, and if the identified position, size, orientation, or a combination of two or more of these is included in the determination range, it is determined that the appropriateness condition is satisfied. (Appendix 11) In the determining step, orthogonally transforming the plurality of key points of the moving object detected from the captured image to calculate key point corresponding points on the orthoimage corresponding to each of the plurality of key points; A keypoint correction method as described in Appendix 10, wherein a position, size, or orientation of the moving object on the orthoimage determined based on the calculated positions of the multiple keypoint corresponding points is calculated as the position, size, or orientation of the moving object. (Appendix 12) a decision range generating step of generating the decision range indicated by the appropriateness condition by using a plurality of teacher images in which the correct key points for the moving object are determined; In the determination range generating step, For each of the plurality of teacher images, an error between the correct key point in the teacher image and the key point of the moving object detected from the teacher image is calculated, and if the error is equal to or smaller than a threshold, a feature vector of the key point of the moving object is calculated; A keypoint correction method as described in Appendix 11, in which the determination range is generated using the feature vector included in a cluster whose stability is equal to or greater than a threshold value among multiple clusters obtained by clustering the calculated multiple feature vectors. (Appendix 13) 10. The keypoint correction method according to claim 9, wherein in the determination step, a time stability index value representing temporal stability is calculated for the multiple keypoints of the moving object, and if the time stability index value is equal to or greater than a threshold value, it is determined that the appropriateness condition is satisfied. (Appendix 14) 14. The keypoint correction method according to claim 13, wherein the time stability index value is larger as the amount of time change in size of the moving object within a predetermined time period is smaller. (Appendix 15) 15. The keypoint correction method according to claim 14, wherein the time stability index value is larger as the reliability of the multiple keypoints of the moving object is higher. (Appendix 16) the correction parameter represents a size of the moving object; A key point correction method according to any one of appendixes 9 to 15, wherein, in the correction step, when the correction parameters have been determined for the moving object for which multiple key points have been detected from the captured image, one or more of the key points are corrected so that the size of the moving object represented by the multiple key points becomes the size represented by the correction parameters. (Appendix 17) On the computer, An acquisition step of acquiring a plurality of captured images; a detection step of detecting a plurality of key points of a moving object from each of the captured images; a determination step of determining whether or not the plurality of key points of the moving object detected from the captured image satisfy a predetermined appropriate condition; a determining step of determining, when it is determined that the plurality of key points of the moving object satisfy the appropriateness condition, a correction parameter to be used for correcting the plurality of key points detected for the moving object, using the plurality of key points; and a correction step of correcting the multiple key points using the correction parameters when the correction parameters have been determined for the moving object from which the multiple key points have been detected in the captured image. (Appendix 18) the appropriateness condition indicates a determination range that is a range that the position, size, orientation, or a combination of two or more of the moving object should satisfy; The program described in Appendix 17, wherein in the determination step, the position, size, or orientation of the moving object is identified using multiple key points of the moving object, and if the identified position, size, orientation, or a combination of two or more of these is included in the determination range, it is determined that the appropriateness condition is satisfied. (Appendix 19) In the determining step, orthogonally transforming the plurality of key points of the moving object detected from the captured image to calculate key point corresponding points on the orthoimage corresponding to each of the plurality of key points; The program described in Appendix 18, which calculates the position, size, or orientation of the moving object on the orthoimage based on the calculated positions of the multiple keypoint corresponding points as the position, size, or orientation of the moving object. (Appendix 20) a decision range generating step of generating the decision range indicated by the appropriateness condition by using a plurality of teacher images in which the correct key points for the moving object are determined; In the determination range generating step, For each of the plurality of teacher images, an error between the correct key point in the teacher image and the key point of the moving object detected from the teacher image is calculated, and if the error is equal to or smaller than a threshold, a feature vector of the key point of the moving object is calculated; The program according to claim 19, further comprising: generating the determination range using the feature vector included in a cluster having a stability equal to or greater than a threshold value among a plurality of clusters obtained by clustering the calculated plurality of feature vectors. (Appendix 21) The program described in Appendix 17, wherein in the judgment step, a time stability index value representing temporal stability is calculated for the multiple key points of the moving object, and if the time stability index value is equal to or greater than a threshold, it is judged that the appropriate condition is satisfied. (Appendix 22) 22. The program according to claim 21, wherein the time stability index value is larger as the amount of time change in size of the moving object within a predetermined time period is smaller. (Appendix 23) 23. The program of claim 22, wherein the time stability index value is larger as the reliability of the multiple key points of the moving object is higher. (Appendix 24) the correction parameter represents a size of the moving object; 24. The program according to any one of appendices 17 to 23, wherein, in the correction step, when the correction parameters have been determined for the moving object for which multiple key points have been detected from the captured image, one or more of the key points are corrected so that the size of the moving object represented by the multiple key points becomes the size represented by the correction parameters. [Explanation of symbols]

[0122] 10. Captured images 20 Target Object 30 Key Points 40 Camera 50 orthoimages 60 Bounding Box 80 keypoint correspondences 90 Long side 100 rectangular area 110 Correction rectangle 120 Vertex 500 Computers 502 Bus 504 Processor 506 Memory 508 Storage Devices 510 Input / Output Interface 512 Network Interface 600 Network 2000 Keypoint Corrector 2020 Acquisition Department 2040 Detector 2060 Judgment section 2080 Decision Section 2100 Correction section 2120 Decision range generation unit

Claims

1. An acquisition unit that acquires a plurality of captured images; A detection unit that detects a plurality of key points of a moving object from each of the captured images; a determination unit that determines whether or not the plurality of key points of the moving object detected from the captured image satisfy a predetermined appropriate condition; a determination unit that, when it is determined that the plurality of key points of the moving object satisfy the appropriateness condition, determines correction parameters to be used for correcting the plurality of key points detected for the moving object, using the plurality of key points; a correction unit that, when the correction parameters have been determined for the moving object from which the plurality of key points have been detected in the captured image, corrects the plurality of key points using the correction parameters; the appropriateness condition indicates a determination range that is a range that the position, size, orientation, or a combination of two or more of the above-mentioned moving object should satisfy; The determination unit uses multiple key points of the moving object to identify the position, size, or orientation of the moving object, and determines that the suitability condition is satisfied if the identified position, size, orientation, or a combination of two or more of these is included in the determination range.

2. The determination unit is orthogonally transforming the plurality of key points of the moving object detected from the captured image to calculate key point corresponding points on the orthoimage corresponding to each of the plurality of key points; The key point correction device according to claim 1 , wherein the position, size, or orientation of the moving object on the orthoimage, which is determined based on the calculated positions of the multiple key point corresponding points, is calculated as the position, size, or orientation of the moving object.

3. a decision range generating unit that generates the decision range indicated by the appropriate condition by using a plurality of teacher images in which the key points of the moving object are determined as correct answers; The determination range generation unit For each of the plurality of teacher images, an error between the correct key point in the teacher image and the key point of the moving object detected from the teacher image is calculated, and if the error is equal to or smaller than a threshold, a feature vector of the key point of the moving object is calculated; The keypoint correction device according to claim 2 , wherein the determination range is generated using the feature vectors included in a cluster having a stability equal to or greater than a threshold value among a plurality of clusters obtained by clustering the calculated plurality of feature vectors.

4. An acquisition unit that acquires a plurality of captured images; A detection unit that detects a plurality of key points of a moving object from each of the captured images; a determination unit that determines whether or not the plurality of key points of the moving object detected from the captured image satisfy a predetermined appropriate condition; a determination unit that, when it is determined that the plurality of key points of the moving object satisfy the appropriateness condition, determines correction parameters to be used for correcting the plurality of key points detected for the moving object, using the plurality of key points; a correction unit that, when the correction parameters have been determined for the moving object from which the plurality of key points have been detected in the captured image, corrects the plurality of key points using the correction parameters; The determination unit calculates a time stability index value representing temporal stability for the multiple key points of the moving object, and determines that the appropriateness condition is satisfied if the time stability index value is greater than or equal to a threshold value.

5. The keypoint correction device according to claim 4 , wherein the time stability index value is larger as the amount of time change in size of the moving object within a given time period is smaller.

6. The keypoint correction device according to claim 5 , wherein the time stability index value is smaller as the reliability of the plurality of keypoints of the moving object is higher.

7. the correction parameter represents a size of the moving object; 7. The key point correction device according to claim 1, wherein, when the correction parameters have been determined for the moving object for which a plurality of the key points have been detected from the captured image, the correction unit corrects one or more of the key points so that a size of the moving object represented by the plurality of the key points becomes a size represented by the correction parameters.

8. 1. A computer implemented method for keypoint correction, comprising: An acquisition step of acquiring a plurality of captured images; a detection step of detecting a plurality of key points of a moving object from each of the captured images; a determination step of determining whether or not the plurality of key points of the moving object detected from the captured image satisfy a predetermined appropriate condition; a determining step of determining, when it is determined that the plurality of key points of the moving object satisfy the appropriateness condition, a correction parameter to be used for correcting the plurality of key points detected for the moving object, using the plurality of key points; a correction step of correcting the plurality of key points using the correction parameters when the correction parameters have been determined for the moving object from which the plurality of key points have been detected in the captured image; the appropriateness condition indicates a determination range that is a range that the position, size, orientation, or a combination of two or more of the above-mentioned moving object should satisfy; A key point correction method in which, in the determination step, the position, size, or orientation of the moving object is identified using multiple key points of the moving object, and if the identified position, size, orientation, or a combination of two or more of these is included in the determination range, it is determined that the appropriateness condition is satisfied.

9. A computer-implemented method for keypoint correction, comprising: An acquisition step of acquiring a plurality of captured images; a detection step of detecting a plurality of key points of a moving object from each of the captured images; a determination step of determining whether or not the plurality of key points of the moving object detected from the captured image satisfy a predetermined appropriate condition; a determining step of determining, when it is determined that the plurality of key points of the moving object satisfy the appropriateness condition, a correction parameter to be used for correcting the plurality of key points detected for the moving object, using the plurality of key points; a correction step of correcting the plurality of key points using the correction parameters when the correction parameters have been determined for the moving object from which the plurality of key points have been detected in the captured image; A keypoint correction method in which, in the determination step, a time stability index value representing temporal stability is calculated for each of the multiple keypoints of the moving object, and if the time stability index value is equal to or greater than a threshold value, it is determined that the appropriate condition is satisfied.

10. On the computer, An acquisition step of acquiring a plurality of captured images; a detection step of detecting a plurality of key points of a moving object from each of the captured images; a determination step of determining whether or not the plurality of key points of the moving object detected from the captured image satisfy a predetermined appropriate condition; a determining step of determining, when it is determined that the plurality of key points of the moving object satisfy the appropriateness condition, a correction parameter to be used for correcting the plurality of key points detected for the moving object, using the plurality of key points; a correction step of correcting the plurality of key points using the correction parameters when the correction parameters have been determined for the moving object from which the plurality of key points have been detected in the captured image; the appropriateness condition indicates a determination range that is a range that the position, size, orientation, or a combination of two or more of the above-mentioned moving object should satisfy; A program in which, in the determination step, the position, size, or orientation of the moving object is identified using multiple key points of the moving object, and if the identified position, size, orientation, or a combination of two or more of these is included in the determination range, it is determined that the appropriateness condition is satisfied.

Images