Information processing method, device, and program

Abstract

This information processing device estimates three-dimensional posture information on an object on the basis of a camera parameter and two-dimensional posture information on the object within each 2D-BBOX (420, 421) detected using a detector from images (400, 401) included in a multi-view image, calculates a refined 2D-BBOX range from two-dimensional posture information on the object obtained by reprojecting the estimated three-dimensional position information on each image (400, 401, 402), and generates each calculated 2D-BBOX range as a pseudo-label (440, 441, 442).

Description

Information processing method, device, and program

[0001] The disclosed technology relates to an information processing method, an information processing device, and an information processing program.

[0002] Conventionally, object detection has been performed from multi-viewpoint images in which an object is captured from multiple different viewpoints. For example, an image monitoring device has been proposed that captures images of a person in different background images by using a ceiling camera and a wall camera to capture images from different directions. This device projects pixels of a change area in the wall camera image onto the ceiling camera image to obtain an epipolar line, extracts areas of the epipolar line that share characteristics with pixels of the change area, and generates a projected area based on the area where the area exists. This device also combines the projected area with the change area in the ceiling camera image to obtain a combined change area, and detects the person in the ceiling camera image based on the combined change area.

[0003] Machine learning models, such as neural networks, are used to detect objects in images. To execute machine learning using such machine learning models, a large amount of data with correct labels indicating the position information of objects in the image is required. However, preparing a large amount of data with correct labels requires a huge amount of work. Therefore, a semi-supervised learning method has been proposed, in which the position information of objects detected by the machine learning model is used as pseudo labels, and the machine learning of the machine learning model is executed using the pseudo-labeled data in addition to the previously prepared data with correct labels.

[0004] JP 2010-045501 A

[0005] As described above, in semi-supervised learning, when the position information of an object detected by a machine learning model is used as a pseudo label, if the accuracy of the position information of the detected object is low, the accuracy of the machine learning model that performs machine learning using the pseudo label also decreases. In particular, when the object can assume a variety of postures, such as a gymnast, it is difficult to accurately detect the position information of the object from an image.

[0006] In one aspect, the disclosed technology aims to accurately calculate position information of an object in an image.

[0007] In one aspect, the disclosed technology acquires multiple images of an object captured by multiple cameras that capture the object from multiple different viewpoints. The disclosed technology also estimates three-dimensional position information of the object based on two-dimensional position information of the object detected from each of the multiple images and camera parameters of each of the multiple cameras. The disclosed technology then projects the three-dimensional position information of the object onto at least one of the multiple images based on the camera parameters of the camera that captured the image, and calculates the two-dimensional position information of the object in the at least one image.

[0008] As one aspect, it has an effect of being able to calculate position information of an object in an image with high accuracy.

[0009] 1 is a schematic diagram showing a connection between an information processing device and a camera according to the present embodiment. FIG. 1 is a diagram for explaining machine learning of a detector that detects a 2D-BBOX, and detection of the 2D-BBOX. FIG. 2 is a diagram for explaining machine learning of a detector using semi-supervised learning. FIG. 3 is a functional block diagram of an information processing device according to the present embodiment. FIG. 4 is a diagram for explaining 2D-BBOX. FIG. 5 is a diagram for explaining two-dimensional pose information of an object. FIG. 6 is a diagram for explaining projection of three-dimensional pose information of an object onto an image, and calculation of two-dimensional pose information of the object. FIG. 7 is a diagram for explaining the effect of calculating two-dimensional pose information by projecting three-dimensional pose information. FIG. 8 is a diagram for explaining selection of pseudo labels based on spatial constraints. FIG. 9 is a diagram for explaining selection of pseudo labels based on temporal constraints. FIG. 10 is a diagram for explaining selection based on evaluation of pseudo labels. FIG. 11 is a diagram showing a schematic configuration of a computer that functions as an information processing device according to the present embodiment. FIG. 12 is a flowchart showing an example of information processing according to the present embodiment. FIG. 13 is a diagram showing an example of pseudo label generation results by the information processing device according to the present embodiment. FIG. 14 is a diagram for explaining application of the information processing device according to the present embodiment to a judging system for gymnastics.

[0010] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0011] As shown in Fig. 1, an information processing device 10 according to this embodiment is connected to a plurality of cameras 30n that capture images of an object 90 (a person in the example of Fig. 1) at viewpoint n from each of different directions. In the example of Fig. 1, n = 0, 1, 2, and a camera 300 that captures images from viewpoint 0, a camera 301 that captures images from viewpoint 1, and a camera 302 that captures images from viewpoint 2 are connected to the information processing device 10. Note that the number of cameras 30n connected to the information processing device 10 is not limited to the example of Fig. 1, and may be two, four or more.

[0012] The camera 30n is installed at an angle and position such that the target object 90 falls within the shooting range. The images captured by the camera 30n are sequentially input to the information processing device 10. A synchronization signal is sent to each camera 30n, and the images captured by each camera 30n are synchronized.

[0013] The information processing device 10 calculates refined two-dimensional position information of the object 90 based on the two-dimensional position information of the object 90 detected from each of multiple images taken from multiple different viewpoints (hereinafter referred to as ``multi-viewpoint images'').

[0014] Here, a detector that is a machine learning model such as a neural network is used to detect the object 90 from each image. This detector is generated by machine learning using a large number of images with correct labels indicating the two-dimensional position information of the object 90, as shown in the upper diagram of FIG. 2, for example. In the example of FIG. 2, the coordinates [x １ , y １ ] and the coordinates of the bottom right point [x ２ , y ２ ] is used as the correct label. When an unlabeled image is input to a detector that has undergone machine learning using such an image with a correct label, a 2D-BBOX indicating the position of the object 90 is detected from the image, as shown in the lower diagram of FIG.

[0015] As described above, preparing a large number of images with correct answers to perform machine learning of a detector requires a huge amount of work. Therefore, it is possible to perform machine learning of a detector using semi-supervised learning as shown in FIG. 3. Specifically, in semi-supervised learning, unlabeled images are input to a detector that has undergone machine learning using images with correct answers ("Machine Learning 1" in FIG. 3), and the resulting detection results are generated as pseudo labels. Then, the generated pseudo labels are assigned to the original images, and machine learning of the detector is performed using pseudo-labeled images ("Machine Learning 2" in FIG. 3). This allows a large number of pseudo-labeled images to be used for machine learning, making it possible to perform machine learning of a detector even when there are few images with correct answers.

[0016] However, when using the detection results of a single image as pseudo labels, it is difficult to reduce the number of false positive and false negative pseudo labels. Furthermore, even pseudo labels that are neither false positive nor false negative may have a positional bias problem. Positional bias refers to a deviation, such as a shift in the position of the area indicated by the pseudo label relative to the actual area of ​​the object 90 in the image, or a deviation such as a large or small area. Furthermore, particularly when the object 90 can assume a variety of postures, such as a gymnast, it is difficult to generate pseudo labels that are balanced for all postures. For example, there may be too many pseudo labels generated from detection results that detect postures of the gymnast simply standing compared to the number of pseudo labels generated from detection results that detect postures of the gymnast performing. In this case, it cannot be said that balanced pseudo labels have been generated in accordance with the diversity of postures of the gymnast.

[0017] It is also possible to use multi-view images and select highly reliable pseudo labels from the generated pseudo labels based on the correspondence between images included in the multi-view images. However, in this case, since the pseudo labels are selected from the detected 2D-BBOX, it is possible to reduce false positive pseudo labels, but it is not possible to reduce false negative pseudo labels. In addition, the above-mentioned problem of position bias and the problem of not being able to generate balanced pseudo labels corresponding to the diversity of postures of the object 90 remain.

[0018] Therefore, in this embodiment, two-dimensional position information of the object 90 in the image is calculated with high accuracy so that false positive and false negative pseudo labels can be reduced. Also, in this embodiment, pseudo labels are generated in which the position bias with respect to the actual area of ​​the object 90 is corrected. Furthermore, in this embodiment, balanced pseudo labels are generated in accordance with the diversity of postures of the object 90. The information processing device 10 according to this embodiment will be described in detail below.

[0019] As shown in FIG. 4, the information processing device 10 functionally includes an acquisition unit 11, an estimation unit 12, a generation unit 13, a selection unit 14, and a machine learning unit 15. A detector 22 and a camera parameter database (DB) 24 are stored in a predetermined storage area of ​​the information processing device 10. The detector 22 is a machine learning model for detecting a 2D-BBOX indicating the area of ​​an object 90 from an image, generated by machine learning using images with correct answers as training data. The camera parameter DB 24 stores internal parameters and external parameters of each camera 30n. The generation unit 13 is an example of a "calculation unit" of the disclosed technology.

[0020] The acquisition unit 11 acquires time-series multi-viewpoint images captured by a plurality of cameras 30n.

[0021] The estimation unit 12 estimates the three-dimensional position information of the object 90 based on a 2D-BBOX indicating the area of ​​the object 90 detected from each image included in the multi-view image and the camera parameters of each camera that captured each image.

[0022] Specifically, as shown in FIG. 5 , the estimation unit 12 uses the detector 22 to detect a 2D-BBOX 42 n indicating the region of the object 90 from an image 40 n captured by a camera 30 n. Then, the estimation unit 12 estimates two-dimensional position information of each part of the object 90 using a recognition model (not shown) previously generated by machine learning to recognize one or more parts of the person who is the object 90 from the detected 2D-BBOX 42 n. For example, as shown in FIG. 6 , when the positions of each joint, etc. of the person who is the object 90 (black circles in FIG. 6 ) are recognized by the recognition model, the estimation unit 12 estimates the coordinate values ​​of the positions of each joint, etc. as two-dimensional position information of the object 90. Hereinafter, a group of two-dimensional position information of each part, such as the joints, of the object 90 is referred to as two-dimensional posture information.

[0023] Furthermore, the estimation unit 12 estimates three-dimensional position information of each part of the object 90 by triangulation using the camera parameters of the camera 30n stored in the camera parameter DB 24 and the estimated two-dimensional posture information of the object 90. Hereinafter, a group of three-dimensional position information of each part of the object 90, such as joints, is referred to as three-dimensional posture information. When n parts such as joints are recognized for one person, which is the object 90, by the recognition model, the three-dimensional posture information is calculated as {[P Ｘ １ , P Ｙ １ , P Ｚ １ ], [P Ｘ ２ , P Ｙ ２ , P Ｚ ２ ], ..., [P Ｘ ｎ , P Ｙ ｎ , P Ｚ ｎ ].

[0024] As shown in A of Fig. 7, the generation unit 13 projects three-dimensional orientation information of the object 90 onto the image 40n based on the camera parameters of the camera 30n that captured the image 40n included in the multi-view image, and calculates refined two-dimensional orientation information of the object 90 in the image 40n. Specifically, the three-dimensional orientation information {[P Ｘ１ , P Ｙ １ , P Ｚ １ ], [P Ｘ ２ , P Ｙ ２ , P Ｚ ２ ], ..., [P Ｘ ｎ , P Ｙ ｎ , P Ｚ ｎ ]} is expressed as two-dimensional posture information corresponding to {[p ｘ １ , p ｙ １ ], [p ｘ ２ , p ｙ ２ ], ..., [p ｘ ｎ , p ｙ ｎ In this case, the generation unit 13 calculates two-dimensional orientation information using the following equation (1). In equation (1), H is a projection matrix from three dimensions to two dimensions, which is determined from the camera parameters of the camera 30 n.

[0025]

[0026] 7B, the generation unit 13 generates a pseudo label 44n indicating the region of the object 90 based on the calculated two-dimensional orientation information. Specifically, as shown in the following equation (2), the generation unit 13 calculates the coordinates [x １ , y １ ], and the coordinates of the bottom right point [x ２ , y ２ ] is calculated.

[0027]

[0028] In addition, w and h in the formula (2) are the width and height of the circumscribing rectangle of the object 90 indicated by the calculated two-dimensional orientation information. １ , y １ ] and [x ２ , y ２In calculating [ ], a value obtained by multiplying the width w or height h by a constant α (for example, α = 0.05) is subtracted or added to calculate an area in which a predetermined margin is provided to a circumscribed rectangle of w × h as the range of the pseudo label 44n. Note that the margin is not limited to the value obtained by multiplying the width w or height h by the constant α. A predetermined number of pixels (for example, 5 pixels) may be used as a margin, and the range obtained by adding a predetermined number of pixels to the top, bottom, left, and right of the range of w × h may be used as the range of the pseudo label 44n.

[0029] In this way, by estimating three-dimensional pose information from the two-dimensional pose information of each image 40n and reprojecting the three-dimensional pose information onto each image 40n to calculate refined two-dimensional pose information, the accuracy of generating pseudo labels can be improved. For example, as shown in FIG. 8, assume that a multi-viewpoint image includes images 400, 401, and 402, and the estimation unit 12 detects 2D-BBOXes 420 and 421 from images 400 and 401, but does not detect 2D-BBOX 422 from image 402. Even in this case, the generation unit 13 can reproject the three-dimensional pose information onto image 402 to generate a pseudo label 442 from image 402. In other words, false negative pseudo labels can be reduced.

[0030] Furthermore, the generation unit 13 can correct the positional bias occurring in the 2D-BBOXes 420 and 421 by re-projecting three-dimensional posture information onto the images 400 and 401 to generate pseudo labels 440 and 441.

[0031] The selector 14 selects pseudo labels to be used for machine learning of the detector 22 from the pseudo labels 44n generated by the generator 13 based on spatial and temporal constraints.

[0032] Specifically, the selection unit 14 selects the pseudo label 44n when the position in three-dimensional space of the object 90 indicated by the three-dimensional posture information that is the projection source when generating the pseudo label 44n (hereinafter referred to as the "three-dimensional position") is included in a predetermined range. For example, if the object 90 is a gymnast, the predetermined range may be a competition area according to the sport. More specifically, in the case of an event using equipment, a predetermined range including the equipment may be defined as the competition area, and in the case of a floor event, a predetermined range including a defined performance area may be defined as the competition area.

[0033] For example, as shown in FIG. 9 , assume that the three-dimensional positions 46A from which the pseudo labels 440A generated from image 400 and 441A generated from image 401 are projected are within the competition area. In this case, the selection unit 14 selects the pseudo labels 440A and 441A as the pseudo labels 44n to be used in machine learning. On the other hand, assume that the three-dimensional positions 46B from which the pseudo labels 440B generated from image 400 and 441B generated from image 401 are projected are outside the competition area. In this case, the selection unit 14 excludes the pseudo labels 440B and 441B from the pseudo labels 44n to be used in machine learning. This makes it possible to exclude the pseudo labels 44n generated for assistants, referees, and other people other than players who have been mistakenly detected.

[0034] Furthermore, if the photographing time of an image for which a pseudo label 44 n has been generated falls within a predetermined time range, the selection unit 14 selects the pseudo label 44 n as the pseudo label 44 n to be used in machine learning. For example, if the object 90 is a gymnast, the predetermined time range may be the time range from the start to the end of the gymnast's performance.

[0035] More specifically, as shown in FIG. 10 , the selection unit 14 identifies a start frame corresponding to the start of a performance and an end frame corresponding to the end of a performance from each frame of the time-series multi-viewpoint images. In the case of an event using equipment, for example, the selection unit 14 identifies as the start frame a frame that is a predetermined number of frames before the moment when the athlete enters the competition area and their feet first leave the floor. The selection unit 14 also identifies as the end frame a frame that is a predetermined number of frames before the athlete exits the competition area. The selection unit 14 then defines the period from the start frame to the end frame as a target time and selects pseudo labels 44 n generated from frames (images 40 n) included in the target time. Meanwhile, the selection unit 14 excludes pseudo labels 44 n generated from non-target frames outside the target time. This allows the selection of pseudo labels 44 n based on the athlete's posture, such as when they are simply standing before the start of the performance, and allows the selection of balanced pseudo labels 44 n that reflect the diversity of athlete postures.

[0036] The selection unit 14 also evaluates the quality of the generated pseudo labels 44n, and if the evaluation result satisfies a standard, selects the pseudo labels 44n as those to be used in machine learning by the detector 22. Specifically, the selection unit 14 calculates the overlap between the 2D-BBOX 42n detected by the estimation unit 12 using the detector 22 and the pseudo labels 44n generated by the generation unit 13 based on the 2D-BBOX 42n. The overlap may be expressed as, for example, the area of ​​the overlapping portion divided by the area of ​​the pseudo labels 44n. As shown in FIG. 11 , the selection unit 14 selects pseudo labels 44n whose overlap is equal to or greater than a predetermined threshold, and excludes pseudo labels 44n whose overlap is less than the threshold.

[0037] Furthermore, the selection unit 14 may present the pseudo labels 44 n whose overlapping degree is less than a threshold to the user, accept a decision by the user on whether to accept or reject the pseudo labels 44 n, and select the pseudo labels 44 n adopted by the user as the pseudo labels 44 n to be used in the machine learning of the detector 22. This reduces the burden on the user because the user is only required to decide on the pseudo labels 44 n that do not satisfy the criteria, compared to when the user decides on whether to accept or reject all of the generated pseudo labels 44 n.

[0038] The machine learning unit 15 performs machine learning of the detector 22 using pseudo-labeled images, in which the pseudo labels 44n selected by the selection unit 14 are added to the images 40n, and images with correct answers as training data. The machine learning unit 15 repeatedly executes the processes of the acquisition unit 11, the estimation unit 12, the generation unit 13, and the selection unit 14, and repeatedly executes machine learning of the detector 22 using the obtained pseudo labels 44n. By repeating the process, the number of pseudo-labeled images increases, thereby improving the detection accuracy of the 2D-BBOX 42n by the detector 22 and also improving the generation accuracy of the pseudo labels 44n. Furthermore, by using only pseudo labels 44n whose quality evaluation results satisfy the standard by the selection unit 14 during the repeated process, the detection accuracy of the 2D-BBOX 42n by the detector 22 is further improved.

[0039] The information processing device 10 may be realized, for example, by a computer 50 shown in FIG. 12 . The computer 50 includes a CPU (Central Processing Unit) 51, a memory 52 as a temporary storage area, and a non-volatile storage device 53. The computer 50 also includes an input / output device 54 such as an input device and a display device, and an R / W (Read / Write) device 55 that controls reading and writing of data from and to a storage medium 59. The computer 50 also includes a communication I / F (Interface) 56 that is connected to a network such as the Internet. The CPU 51, memory 52, storage device 53, input / output device 54, R / W device 55, and communication I / F 56 are connected to one another via a bus 57.

[0040] The storage device 53 is, for example, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The storage device 53 serving as a storage medium stores an information processing program 60 for causing the computer 50 to function as the information processing device 10. The information processing program 60 includes an acquisition process control instruction 61, an estimation process control instruction 62, a generation process control instruction 63, a selection process control instruction 64, and a machine learning process control instruction 65. The storage device 53 also includes an information storage area 70 in which information constituting the detector 22 and the camera parameter DB 24 is stored.

[0041] The CPU 51 reads the information processing program 60 from the storage device 53, loads it into the memory 52, and sequentially executes the control instructions contained in the information processing program 60. The CPU 51 operates as the acquisition unit 11 shown in FIG. 4 by executing the acquisition process control instruction 61. The CPU 51 operates as the estimation unit 12 shown in FIG. 4 by executing the estimation process control instruction 62. The CPU 51 operates as the generation unit 13 shown in FIG. 4 by executing the generation process control instruction 63. The CPU 51 operates as the selection unit 14 shown in FIG. 4 by executing the selection process control instruction 64. The CPU 51 operates as the machine learning unit 15 shown in FIG. 4 by executing the machine learning process control instruction 65. The CPU 51 reads information from the information storage area 70 and loads the detector 22 and the camera parameter DB 24 into the memory 52. ​​As a result, the computer 50 that executes the information processing program 60 functions as the information processing device 10. The CPU 51 that executes the program is hardware.

[0042] The functions realized by the information processing program 60 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0043] Next, the operation of the information processing device 10 according to this embodiment will be described. When time-series multi-viewpoint images are input to the information processing device 10 and an instruction is given to execute machine learning of the detector 22, the information processing device 10 executes the information processing shown in Fig. 13. Note that the information processing is an example of an information processing method of the disclosed technology.

[0044] In step S11, the acquisition unit 11 acquires a plurality of time-series multi-viewpoint images. Next, in step S12, the estimation unit 12 detects a 2D-BBOX 42n indicating the region of the object 90 from each image 40n included in the multi-viewpoint images using the detector 22. Then, the estimation unit 12 estimates two-dimensional posture information of the object 90 from the detected 2D-BBOX 42n using a recognition model. Next, in step S13, the estimation unit 12 estimates three-dimensional posture information of the object 90 by triangulation using the camera parameters of the camera 30n stored in the camera parameter DB 24 and the estimated two-dimensional posture information of the object 90.

[0045] Next, in step S14, the generation unit 13 projects three-dimensional posture information of the object 90 onto each image 40n based on the camera parameters of the camera 30n that captured each image 40n, and calculates two-dimensional posture information of the refined object 90 in each image 40n. Then, the generation unit 13 generates a pseudo label 44n based on the calculated two-dimensional posture information.

[0046] Next, in step S15, the selection unit 14 selects a pseudo label to be used for machine learning of the detector 22 from the pseudo labels 44n generated in step S14 based on spatiotemporal constraints. Specifically, the selection unit 14 selects the pseudo label 44n when the three-dimensional position of the object 90 indicated by the three-dimensional posture information that is the projection source when generating the pseudo label 44n is included in a predetermined range. Furthermore, the selection unit 14 selects the pseudo label 44n when the shooting time of the image for which the pseudo label 44n was generated is included in a predetermined time range.

[0047] Next, in step S16, the selection unit 14 evaluates the quality of the pseudo label 44n selected in step S15, and if the evaluation result satisfies a criterion, selects the pseudo label 44n as the pseudo label 44n to be used in machine learning of the detector 22. Next, in step S17, the machine learning unit 15 executes machine learning of the detector 22 using, as training data, a pseudo-labeled image in which the pseudo label 44n selected in step S16 is added to the image 40n, and an image with a correct answer.

[0048] Next, in step S18, the machine learning unit 15 determines whether or not a termination condition for the machine learning of the detector 22 is satisfied. For example, the machine learning unit 15 determines that the termination condition is satisfied when the number of repetitions reaches a predetermined number, when the detection accuracy of the detector 22 reaches a predetermined value, when the detection accuracy of the detector 22 converges, etc. If the termination condition is not satisfied, the process returns to step S11, and if the termination condition is satisfied, the information processing ends.

[0049] As described above, the information processing device according to this embodiment estimates three-dimensional position information of an object based on two-dimensional position information of the object detected from each image included in a multi-view image and camera parameters. The information processing device then projects the three-dimensional position information of the object onto each image based on the camera parameters, and calculates refined two-dimensional position information of the object. This makes it possible to accurately calculate the position information of the object in the image. Furthermore, by generating pseudo labels based on this two-dimensional position information, it is possible to reduce false negatives of the pseudo labels and correct bias in the positions of the pseudo labels.

[0050] Furthermore, the information processing device according to this embodiment can generate balanced pseudo labels that correspond to the diversity of the object's posture by selecting pseudo labels to be used for machine learning of the detector from the generated pseudo labels based on spatiotemporal constraints.

[0051] Here, Fig. 14 shows an example of the results of generating pseudo labels by the information processing device according to this embodiment. The three diagrams on the left in Fig. 14 are diagrams that schematically show an example of detection results by a method for detecting 2D-BBOX using a detector before applying semi-supervised learning according to this embodiment (hereinafter referred to as the "comparison method"). Also, the three diagrams on the right in Fig. 14 are diagrams that schematically show an example of detection results by a method for detecting 2D-BBOX using a detector to which semi-supervised learning according to this embodiment is applied (hereinafter referred to as the "present method").

[0052] As shown in the top diagram of Figure 14, the inaccurate 2D-BBOX produced by the comparison method is improved by the present method. Also, as shown in the middle diagram of Figure 14, the 2D-BBOX that was missing by the comparison method is detected by the present method. Also, as shown in the bottom diagram of Figure 14, the comparison method erroneously detected 2D-BBOXes that indicated people other than the players, who were the intended targets, but the present method eliminated this erroneous detection.

[0053] The information processing device according to the above embodiment can be applied to, for example, a scoring system for gymnastics. Here, an outline of the processing of the scoring system for gymnastics will be described with reference to FIG.

[0054] When multi-view images are input, the scoring system detects a person's area from each image included in the multi-view images. Next, the scoring system determines whether the person indicated by the detected area is a player or a non-player, based on whether the position where the person is located is within the competition area, etc., and identifies the area representing the player. The scoring system tracks the player by matching areas representing the same player in the time-series multi-view images. The scoring system recognizes the player's two-dimensional skeletal information from each of the tracked series of images using a recognition model, etc. The scoring system estimates three-dimensional skeletal information from the two-dimensional skeletal information using camera parameters. The scoring system then performs post-processing, such as smoothing, on the time-series three-dimensional skeletal information to estimate the phases (breaks) of the performance and then recognizes the technique.

[0055] In the scoring system, a detector that has undergone machine learning using pseudo labels generated by the information processing device according to the embodiment can be applied to the process of detecting a person area.

[0056] In the above embodiment, the three-dimensional posture information, which is the estimated three-dimensional position information, is projected onto all images included in the multi-view image, but this is not limiting. The three-dimensional posture information may be projected onto at least one image of the multi-view image, for example, onto an image in which the 2D-BBOX is not detected by the detector.

[0057] Furthermore, the disclosed technology is not limited to cases where the object is a gymnast, but can be applied to various people as the object, such as athletes of other sports, ordinary pedestrians, etc. Furthermore, it can also be applied to objects other than people, such as animals and vehicles.

[0058] In the above embodiment, the information processing program is stored (installed) in advance in a storage device, but this is not limiting. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0059] 10 Information processing device 11 Acquisition unit 12 Estimation unit 13 Generation unit 14 Selection unit 15 Machine learning unit 22 Detector 24 Camera parameter DB 30n Camera 40n Image 42n 2D-BBOX 44n Pseudo label 50 Computer 51 CPU 52 Memory 53 Storage device 54 Input / output device 55 R / W device 56 Communication I / F 57 Bus 59 Storage medium 60 Information processing program 61 Acquisition process control command 62 Estimation process control command 63 Generation process control command 64 Selection process control command 65 Machine learning process control command 70 Information storage area 90 Object

Claims

1. Acquiring a plurality of images captured by a plurality of cameras that capture an object from a plurality of different viewpoints; estimating three-dimensional position information of the object based on two-dimensional position information of the object detected from each of the plurality of images and camera parameters of each of the plurality of cameras; projecting three-dimensional position information of the object onto at least one image based on camera parameters of a camera that captured at least one image of the plurality of images, and calculating two-dimensional position information of the object in the at least one image; An information processing method in which a computer executes a process including the steps of:

2. The information processing method according to claim 1 , wherein the process of calculating the two-dimensional position information of the object includes calculating information indicating a region of the object.

3. 3. The information processing method according to claim 1 or 2, wherein, when the object is a person, the process of estimating the three-dimensional position information of the object includes detecting two-dimensional posture information of the object from each of the plurality of images as the two-dimensional position information of the object, and estimating the three-dimensional posture information of the object based on the two-dimensional posture information of the object.

4. The information processing method according to claim 1 or claim 2, wherein the computer executes a process including performing machine learning of a machine learning model for detecting the two-dimensional position information of the object from an image, using an image to which the calculated two-dimensional position information of the object is assigned as a pseudo label as training data.

5. The information processing method according to claim 4 , wherein the training data is images to which, among the pseudo labels, a pseudo label is added such that three-dimensional position information of the object from which the projection originates is included within a predetermined range in three-dimensional space.

6. 6. The information processing method according to claim 5, wherein when the object is a gymnast, the predetermined area is a competition area according to the sport.

7. The information processing method according to claim 4 , wherein images to which a pseudo label is assigned whose corresponding image capture time falls within a predetermined time range are set as the training data.

8. 8. The information processing method according to claim 7, wherein when the object is a gymnast, the predetermined time range is a time range corresponding to a start to an end of a performance.

9. The information processing method according to claim 4 , wherein the generation of the pseudo labels and the machine learning of the machine learning model are repeatedly performed using the images to which the pseudo labels have been assigned as training data.

10. The information processing method according to claim 9, further comprising excluding from the training data images to which a pseudo label has been assigned, the degree of overlap between an area indicated by two-dimensional position information of the detected object and an area indicated by the generated pseudo label being less than a predetermined threshold.

11. an acquisition unit that acquires a plurality of images captured by a plurality of cameras that capture an object from a plurality of different viewpoints; an estimation unit that estimates three-dimensional position information of the object based on two-dimensional position information of the object detected from each of the plurality of images and camera parameters of each of the plurality of cameras; a calculation unit that projects three-dimensional position information of the object onto at least one image based on camera parameters of a camera that captured at least one image of the plurality of images, and calculates two-dimensional position information of the object in the at least one image; An information processing device comprising:

12. Acquiring a plurality of images captured by a plurality of cameras that capture an object from a plurality of different viewpoints; estimating three-dimensional position information of the object based on two-dimensional position information of the object detected from each of the plurality of images and camera parameters of each of the plurality of cameras; projecting three-dimensional position information of the object onto at least one image based on camera parameters of a camera that captured at least one image of the plurality of images, and calculating two-dimensional position information of the object in the at least one image; An information processing program for causing a computer to execute processing including the above.