Information processing methods, apparatus, and programs

Abstract

This information processing device estimates three-dimensional posture information on an object on the basis of a camera parameter and two-dimensional posture information on the object within each 2D-BBOX (420, 421) detected using a detector from images (400, 401) included in a multi-view image, calculates a refined 2D-BBOX range from two-dimensional posture information on the object obtained by reprojecting the estimated three-dimensional position information on each image (400, 401, 402), and generates each calculated 2D-BBOX range as a pseudo-label (440, 441, 442).

Description

[Technical Field]

[0001] The disclosed technologies relate to information processing methods, information processing devices, and information processing programs. [Background technology]

[0002] Conventionally, objects have been detected from multi-view images taken from multiple different viewpoints. For example, an image surveillance device has been proposed that acquires images of people in different background images by taking pictures from different directions using a ceiling camera and a wall camera. This device projects pixels of the change region in the wall camera image onto the ceiling camera image to obtain epipolar lines, extracts regions of the epipolar lines that have common features with the pixels of the change region, and generates a projected region based on the region where these regions exist. Furthermore, this device combines the projected region and the change region in the ceiling camera image to obtain a composite change region, and detects people in the ceiling camera image based on this composite change region.

[0003] To detect objects from images, machine learning models such as neural networks are used. Performing machine learning on such models requires a large amount of data with ground truth labels indicating the location of objects within an image. However, preparing such a large amount of ground truth data incurs enormous labor costs. Therefore, a semi-supervised learning method has been proposed that uses the location information of objects detected by the machine learning model as pseudo-labels, and performs machine learning on the machine learning model using both pre-prepared ground truth data and pseudo-labeled data. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2010-045501 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] As described above, in semi-supervised learning, when using the position information of an object detected by a machine learning model as a pseudo label, if the accuracy of the position information of the detected object is low, the accuracy of the machine learning model in which machine learning is executed using the pseudo label will also decrease. In particular, when the object can take various poses, such as a gymnastics athlete, it is difficult to accurately detect the position information of the object from an image.

[0006] As one aspect, the disclosed technology aims to accurately calculate the position information of an object in an image.

Means for Solving the Problem

[0007] As one aspect, the disclosed technology acquires a plurality of images taken by each of a plurality of cameras that photograph an object from different viewpoints. Further, the disclosed technology estimates the three-dimensional position information of the object based on the two-dimensional position information of the object detected from each of the plurality of images and the camera parameters of each of the plurality of cameras. Then, the disclosed technology projects the three-dimensional position information of the object onto at least one of the plurality of images based on the camera parameters of the camera that has taken at least one of the plurality of images, and calculates the two-dimensional position information of the object in the at least one image.

Advantages of the Invention

[0008] As one aspect, it has the effect that the position information of an object in an image can be accurately calculated.

Brief Description of the Drawings

[0009] [Figure 1] It is a schematic diagram showing the connection between the information processing apparatus and the camera according to the present embodiment. [Figure 2] It is a diagram for explaining the machine learning of a detector that detects 2D-BBOX and the detection of 2D-BBOX. [Figure 3]This diagram illustrates the machine learning of a detector using semi-supervised learning. [Figure 4] This is a functional block diagram of the information processing device according to this embodiment. [Figure 5] This is a diagram to explain 2D-BBOX. [Figure 6] This is a diagram used to explain the two-dimensional orientation information of an object. [Figure 7] This diagram illustrates the projection of an object's 3D pose information onto an image, and the calculation of its 2D pose information. [Figure 8] This diagram illustrates the effect of projecting 3D pose information to calculate 2D pose information. [Figure 9] This diagram illustrates the selection of pseudo-labels based on spatial constraints. [Figure 10] This diagram illustrates the selection of pseudo-labels under time constraints. [Figure 11] This diagram illustrates selection based on the evaluation of pseudo-labels. [Figure 12] This figure shows a schematic configuration of a computer that functions as an information processing device according to this embodiment. [Figure 13] This flowchart shows an example of information processing according to this embodiment. [Figure 14] This figure shows an example of the pseudo-label generation result by the information processing device according to this embodiment. [Figure 15] This figure illustrates the application of the information processing device according to this embodiment to a scoring system for gymnastics competitions. [Modes for carrying out the invention]

[0010] An example of an embodiment relating to the disclosed technology will be described below with reference to the drawings.

[0011] As shown in Figure 1, the information processing device 10 according to this embodiment is connected to each of a plurality of cameras 30n that photograph an object (in the example in Figure 1, the object is a person) 90 from viewpoints n from different directions. In the example in Figure 1, n = 0, 1, 2, and the camera 300 that photographs from viewpoint 0, the camera 301 that photographs from viewpoint 1, and the camera 302 that photographs from viewpoint 2 are connected to the information processing device 10. Note that the number of cameras 30n connected to the information processing device 10 is not limited to the example in Figure 1, and may be 2 or 4 or more.

[0012] Camera 30n is positioned at an angle and location such that the object 90 is within its shooting range. The video captured by camera 30n is sequentially input to the information processing device 10. A synchronization signal is sent to each camera 30n to synchronize the video captured by each camera 30n.

[0013] The information processing device 10 calculates refined two-dimensional position information of the object 90 based on the two-dimensional position information of the object 90 detected from each of the multiple images (hereinafter referred to as "multi-view images") taken from multiple different viewpoints.

[0014] Here, to detect the object 90 from each image, a detector, which is a machine learning model such as a neural network, is used. This detector is generated by machine learning using a large number of images with ground truth labels indicating the 2D position information of the object 90, as shown in the upper part of Figure 2. In the example in Figure 2, the ground truth labels are the coordinates [x1, y1] of the top left point and the coordinates [x2, y2] of the bottom right point of the 2D bounding box (hereinafter referred to as "2D-BBOX") surrounding the object 90 (in the example in Figure 2, the gymnast). As shown in the lower part of Figure 2, by inputting an unlabeled image into a detector that has been machine-learned using such ground truth labeled images, the 2D-BBOX indicating the position of the object 90 is detected from that image.

[0015] As mentioned above, preparing a large number of ground truth images to perform machine learning on a detector requires enormous labor costs. Therefore, it is conceivable to perform machine learning on a detector using semi-supervised learning, as shown in Figure 3. Specifically, in semi-supervised learning, a detector that has undergone machine learning on ground truth labeled images ("Machine Learning 1" in Figure 3) is input with an unlabeled image, and the detection results obtained are generated as pseudo-labels. Then, machine learning on the detector ("Machine Learning 2" in Figure 3) is performed on the pseudo-labeled image obtained by attaching the generated pseudo-labels to the original image. This allows a large number of pseudo-labeled images to be used for machine learning, and enables machine learning on the detector even when there are few ground truth images.

[0016] However, when using the detection results of a single image as false labels, it is difficult to reduce false positives and false negatives. Furthermore, even false labels that are not false positives or false negatives may have positional bias issues. Positional bias refers to discrepancies such as the position of the area indicated by the false label being shifted relative to the actual area of ​​the object 90 in the image, or the size of the area being larger or smaller. Moreover, especially when the object 90 can assume a variety of postures, such as a gymnast, it is difficult to generate a balanced false label for all postures. For example, there may be too many false labels generated from detection results that detect a gymnast's posture during a performance compared with the number of false labels generated from detection results that detect a gymnast simply standing. In this case, it cannot be said that a balanced false label has been generated that corresponds to the diversity of the gymnast's postures.

[0017] Another approach is to use multi-view images and select highly reliable pseudo-labels from the generated pseudo-labels based on the correspondence between images contained in the multi-view images. However, in this case, since pseudo-labels are selected from the detected 2D-BBOX, it is possible to reduce false-positive pseudo-labels, but not false-negative pseudo-labels. Furthermore, the aforementioned positional bias problem and the problem of not being able to generate well-balanced pseudo-labels that account for the diversity of the object 90's posture remain.

[0018] Therefore, in this embodiment, the two-dimensional positional information of the object 90 in the image is calculated with high accuracy so as to reduce false positive and false negative pseudolabels. Furthermore, in this embodiment, pseudolabels are generated in which the bias of the object 90's position relative to its actual area is corrected. Moreover, in this embodiment, balanced pseudolabels are generated according to the diversity of the object 90's posture. The information processing device 10 according to this embodiment will now be described in detail.

[0019] As shown in Figure 4, the information processing device 10 functionally includes an acquisition unit 11, an estimation unit 12, a generation unit 13, a selection unit 14, and a machine learning unit 15. Furthermore, a detector 22 and a camera parameter database 24 are stored in a predetermined storage area of ​​the information processing device 10. The detector 22 is a machine learning model generated by machine learning using ground truth images as training data, for detecting 2D-BBOXes that indicate the region of an object 90 from an image. The camera parameter database 24 stores the internal and external parameters of each camera 30n. Note that the generation unit 13 is an example of the "calculation unit" in the disclosed technology.

[0020] The acquisition unit 11 acquires time-series multi-viewpoint images captured by multiple cameras 30n.

[0021] The estimation unit 12 estimates the three-dimensional position information of the object 90 based on a 2D-BBOX indicating the region of the object 90 detected from each image included in the multi-view image, and the camera parameters of each camera that captured each image.

[0022] Specifically, as shown in FIG. 5, the estimation unit 12 uses the detector 22 to detect a 2D-BBOX 42n indicating the region of the target object 90 from the image 40n captured by the camera 30n. Then, the estimation unit 12 uses a recognition model (not shown) generated in advance by machine learning to estimate the two-dimensional position information of each part of the target object 90 in order to recognize one or more parts of the person who is the target object 90 from the detected 2D-BBOX 42n. For example, as shown in FIG. 6, when the position of each joint or the like of the person who is the target object 90 (black circles in FIG. 6) is recognized by the recognition model, the estimation unit 12 estimates the coordinate values of the positions of each joint or the like as the two-dimensional position information of the target object 90. Hereinafter, a group of two-dimensional position information of each part such as a joint of the target object 90 is referred to as two-dimensional pose information.

[0023] Further, the estimation unit 12 uses the camera parameters of the camera 30n stored in the camera parameter DB 24 and the estimated two-dimensional pose information of the target object 90 to estimate the three-dimensional position information of each part of the target object 90 by triangulation. Hereinafter, a group of three-dimensional position information of each part such as a joint of the target object 90 is referred to as three-dimensional pose information. When n parts such as joints of one person who is the target object 90 are recognized by the recognition model, the three-dimensional pose information is expressed as {[P X 1 , P Y 1 , P Z 1 , [P X 2 , P Y 2 , P Z 2 , ···, [P X n , P Y n , P Z n}.

[0024] As shown in Figure 7A, the generation unit 13 projects the three-dimensional pose information of the object 90 onto the image 40n based on the camera parameters of the camera 30n that captured the image 40n included in the multi-view image, and calculates the refined two-dimensional pose information of the object 90 in the image 40n. Specifically, the three-dimensional pose information {[P X 1 ,P Y 1 ,P Z 1 ],[P X 2 ,P Y 2 ,P Z 2 ],···,[P X n ,P Y n ,P Z n ]} corresponds to 2D pose information {[p x 1 ,p y 1 ],[p x 2 ,p y 2 ],···,[p x n ,p y n Let ]}. In this case, the generation unit 13 calculates the 2D pose information using the following equation (1). In equation (1), H is the projection matrix from 3D to 2D, which is determined from the camera parameters of camera 30n.

[0025]

number

[0026] As shown in Figure 7B, the generation unit 13 generates a pseudo-label 44n indicating the region of the object 90 based on the calculated two-dimensional orientation information. Specifically, as shown in equation (2) below, the generation unit 13 uses the maximum and minimum values ​​of the two-dimensional coordinates of each point included in the two-dimensional orientation information to calculate the coordinates [x1, y1] of the upper left point and the coordinates [x2, y2] of the lower right point of the pseudo-label 44n.

[0027]

number

[0028] In equation (2), w and h are the width and height of the bounding rectangle of the object 90 indicated by the calculated 2D pose information. In calculating the coordinates [x1, y1] and [x2, y2], the range of the pseudo-label 44n is calculated by subtracting or adding a value obtained by multiplying the width w or height h by a constant α (for example, α = 0.05), thereby adding a predetermined margin to the bounding rectangle of w × h. Note that the margin is not limited to the value obtained by multiplying the width w or height h by the constant α. A predetermined number of pixels (for example, 5 pixels) may be used as a margin, and the range of the pseudo-label 44n may be an area added to the top, bottom and left, and right of the w × h range.

[0029] In this way, by estimating 3D pose information from the 2D pose information of each image 40n, and then reprojecting that 3D pose information onto each image 40n to calculate refined 2D pose information, the accuracy of pseudo-label generation can be improved. For example, as shown in Figure 8, suppose the multi-view image includes images 400, 401, and 402, and the estimation unit 12 detects 2D-BBOX 420 and 421 from images 400 and 401, but does not detect 2D-BBOX 422 from image 402. Even in this case, the generation unit 13 can reproject the 3D pose information onto image 402 and generate pseudo-label 442 from image 402. In other words, the number of false negative pseudo-labels can be reduced.

[0030] Furthermore, the generation unit 13 can correct the positional bias occurring in the 2D-BBOX 420 and 421 by reprojecting the 3D pose information onto the images 400 and 401 to generate pseudo-labels 440 and 441.

[0031] The selection unit 14 selects pseudo-labels 44n generated by the generation unit 13 to be used for machine learning of the detector 22, based on spatial and temporal constraints.

[0032] Specifically, the selection unit 14 selects a pseudo-label 44n if the position of the object 90 in three-dimensional space (hereinafter referred to as "three-dimensional position") indicated by the three-dimensional pose information that is the projection source when generating the pseudo-label 44n falls within a predetermined range. For example, if the object 90 is a gymnast, the predetermined range may be the competition area corresponding to the event. More specifically, in the case of an event using equipment, a predetermined range including the equipment may be defined as the competition area, and in the case of an event on the floor, a predetermined range including the defined performance area may be defined as the competition area.

[0033] For example, as shown in Figure 9, suppose the 3D position 46A from which the pseudo-labels 440A generated from image 400 and 441A generated from image 401 are projected is within the competition area. In this case, the selection unit 14 selects pseudo-labels 440A and 441A as pseudo-labels 44n to be used for machine learning. On the other hand, suppose the 3D position 46B from which the pseudo-labels 440B generated from image 400 and 441B generated from image 401 are projected is outside the competition area. In this case, the selection unit 14 excludes pseudo-labels 440B and 441B from the pseudo-labels 44n to be used for machine learning. This makes it possible to exclude pseudo-labels 44n generated for people other than players, such as assistants and referees, if they are mistakenly detected.

[0034] Furthermore, the selection unit 14 selects a pseudo-label 44n to be used for machine learning if the time the image in which the pseudo-label 44n was generated falls within a predetermined time range. For example, if the object 90 is a gymnast, the predetermined time range may be the time range corresponding to the start and end of the performance.

[0035] More specifically, as shown in Figure 10, the selection unit 14 identifies a start frame corresponding to the start of a performance and an end frame corresponding to the end of a performance from each frame of a series of multi-view images in a time series. In the case of events using equipment, the selection unit 14 identifies, for example, the frame a predetermined before the moment when the athlete enters the competition area and their feet first leave the floor as the start frame. The selection unit 14 also identifies the frame a predetermined before the athlete leaves the competition area as the end frame. The selection unit 14 then defines the period from the start frame to the end frame as the target time and selects pseudo-labels 44n generated from frames (images 40n) included in the target time. On the other hand, the selection unit 14 excludes pseudo-labels 44n generated from frames outside the target time. This makes it possible to exclude pseudo-labels 44n based on the posture of an athlete who is simply standing before the start of a performance, and to select a balanced set of pseudo-labels 44n that correspond to the diversity of athlete postures.

[0036] Furthermore, the selection unit 14 evaluates the quality of the generated pseudo-labels 44n, and if the evaluation result meets the criteria, it selects them as pseudo-labels 44n to be used for machine learning of the detector 22. Specifically, the selection unit 14 calculates the degree of overlap between the 2D-BBOX 42n detected by the detector 22 in the estimation unit 12 and the pseudo-labels 44n generated by the generation unit 13 based on the 2D-BBOX 42n. The degree of overlap may be, for example, the area of ​​the overlapping portion divided by the area of ​​the pseudo-label 44n. As shown in Figure 11, the selection unit 14 selects pseudo-labels 44n whose degree of overlap is equal to or greater than a predetermined threshold, and excludes pseudo-labels 44n whose degree of overlap is less than the threshold.

[0037] Alternatively, the selection unit 14 may present pseudo-labels 44n with a repetition rate below a threshold to the user, accept the user's decision on whether to accept or reject them, and select the pseudo-labels 44n accepted by the user as the pseudo-labels 44n to be used for machine learning of the detector 22. This reduces the user's burden compared to when the user makes the decision on whether to accept or reject all of the generated pseudo-labels 44n, as the user is only required to make a decision on pseudo-labels 44n that do not meet the criteria.

[0038] The machine learning unit 15 uses the pseudo-labeled images, which are images 40n to which the pseudo-labels 44n selected by the selection unit 14 have been added, and the ground truth images as training data to perform machine learning on the detector 22. The machine learning unit 15 repeatedly executes the processes of the acquisition unit 11, estimation unit 12, generation unit 13, and selection unit 14, and repeatedly performs machine learning on the detector 22 using the obtained pseudo-labels 44n. As the process is repeated, the number of pseudo-labeled images increases, improving the detection accuracy of the 2D-BBOX 42n by the detector 22 and also improving the generation accuracy of the pseudo-labels 44n. Furthermore, by using only the pseudo-labels 44n whose quality evaluation results meet the criteria by the selection unit 14 during the repeated processing, the detection accuracy of the 2D-BBOX 42n by the detector 22 is further improved.

[0039] The information processing device 10 may be implemented, for example, by a computer 50 as shown in Figure 12. The computer 50 includes a CPU (Central Processing Unit) 51, a memory 52 as a temporary storage area, and a non-volatile storage device 53. The computer 50 also includes input / output devices 54 such as input devices and display devices, and an R / W (Read / Write) device 55 that controls the reading and writing of data to and from the storage medium 59. The computer 50 also includes a communication interface 56 that connects to a network such as the Internet. The CPU 51, memory 52, storage device 53, input / output devices 54, R / W device 55, and communication interface 56 are connected to each other via a bus 57.

[0040] The storage device 53 is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or flash memory. The storage device 53 stores an information processing program 60 that causes the computer 50 to function as an information processing device 10. The information processing program 60 includes an acquisition process control instruction 61, an estimation process control instruction 62, a generation process control instruction 63, a selection process control instruction 64, and a machine learning process control instruction 65. The storage device 53 also has an information storage area 70 in which information constituting the detector 22 and the camera parameter DB 24 is stored.

[0041] The CPU 51 reads the information processing program 60 from the storage device 53, loads it into memory 52, and sequentially executes the control instructions contained in the information processing program 60. By executing the acquisition process control instruction 61, the CPU 51 operates as the acquisition unit 11 shown in Figure 4. The CPU 51 also operates as the estimation unit 12 shown in Figure 4 by executing the estimation process control instruction 62. The CPU 51 also operates as the generation unit 13 shown in Figure 4 by executing the generation process control instruction 63. The CPU 51 also operates as the selection unit 14 shown in Figure 4 by executing the selection process control instruction 64. The CPU 51 also operates as the machine learning unit 15 shown in Figure 4 by executing the machine learning process control instruction 65. The CPU 51 also reads information from the information storage area 70 and loads the detector 22 and camera parameter DB 24 into memory 52. ​​As a result, the computer 50 that executed the information processing program 60 functions as the information processing device 10. Note that the CPU 51 that executes the program is hardware.

[0042] The functions implemented by the information processing program 60 may be implemented, for example, by semiconductor integrated circuits, more specifically by ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), etc.

[0043] Next, the operation of the information processing device 10 according to this embodiment will be described. When a time-series multi-view image is input to the information processing device 10 and the machine learning operation of the detector 22 is instructed, the information processing device 10 executes the information processing shown in Figure 13. Note that the information processing is an example of the information processing method of the disclosed technology.

[0044] In step S11, the acquisition unit 11 acquires multiple time-series multi-view images. Next, in step S12, the estimation unit 12 uses the detector 22 to detect a 2D-BBOX 42n indicating the region of the object 90 from each image 40n included in the multi-view images. Then, the estimation unit 12 uses a recognition model to estimate the two-dimensional pose information of the object 90 from the detected 2D-BBOX 42n. Next, in step S13, the estimation unit 12 uses the camera parameters of the camera 30n stored in the camera parameter DB 24 and the estimated two-dimensional pose information of the object 90 to estimate the three-dimensional pose information of the object 90 by triangulation.

[0045] Next, in step S14, the generation unit 13 projects the three-dimensional pose information of the object 90 onto each image 40n based on the camera parameters of the camera 30n that captured each image 40n, and calculates the refined two-dimensional pose information of the object 90 in each image 40n. Then, the generation unit 13 generates a pseudo-label 44n based on the calculated two-dimensional pose information.

[0046] Next, in step S15, the selection unit 14 selects a pseudo-label to be used for machine learning of the detector 22 from the pseudo-labels 44n generated in step S14, based on spatiotemporal constraints. Specifically, the selection unit 14 selects a pseudo-label 44n if the 3D position of the object 90 indicated by the 3D pose information that is the projection source when generating the pseudo-label 44n falls within a predetermined range. The selection unit 14 also selects a pseudo-label 44n if the time the image in which the pseudo-label 44n was generated was captured falls within a predetermined time range.

[0047] Next, in step S16, the selection unit 14 evaluates the quality of the pseudo-labels 44n selected in step S15, and if the evaluation result meets the criteria, it selects them as pseudo-labels 44n to be used for machine learning of the detector 22. Next, in step S17, the machine learning unit 15 uses the pseudo-labeled image obtained by adding the pseudo-labels 44n selected in step S16 to the image 40n, and the ground truth image, as training data to perform machine learning of the detector 22.

[0048] Next, in step S18, the machine learning unit 15 determines whether the termination conditions for machine learning of the detector 22 are met. For example, the machine learning unit 15 determines that the termination conditions are met when the number of iterations reaches a predetermined number, when the detection accuracy of the detector 22 reaches a predetermined value, or when the detection accuracy of the detector 22 converges. If the termination conditions are not met, the process returns to step S11; if the termination conditions are met, the information processing ends.

[0049] As described above, the information processing device according to this embodiment estimates the three-dimensional position information of an object based on the two-dimensional position information of the object detected from each image included in the multi-view image and camera parameters. Then, the information processing device projects the three-dimensional position information of the object onto each image based on the camera parameters and calculates the refined two-dimensional position information of the object. This makes it possible to calculate the position information of the object in the image with high accuracy. Furthermore, by generating false labels based on this two-dimensional position information, false negatives of false labels can be reduced and the bias of the position of the false labels can be corrected.

[0050] Furthermore, the information processing device according to this embodiment can generate well-balanced pseudo-labels that correspond to the diversity of object orientations by selecting pseudo-labels to be used for machine learning of the detector from the generated pseudo-labels based on spatiotemporal constraints.

[0051] Here, Figure 14 shows an example of the pseudo-label generation result by the information processing device according to this embodiment. The three figures on the left in Figure 14 schematically show an example of the detection result by the method of detecting 2D-BBOX with a detector before applying semi-supervised learning in this embodiment (hereinafter referred to as the "comparison method"). The three figures on the right in Figure 14 schematically show an example of the detection result by the method of detecting 2D-BBOX with a detector to which semi-supervised learning in this embodiment has been applied (hereinafter referred to as the "this method").

[0052] As shown in the upper part of Figure 14, the 2D-BBOX, which was inaccurate in the comparison method, is improved in this method. Also, as shown in the middle part of Figure 14, the 2D-BBOX, which was missing in the comparison method, is detected in this method. Furthermore, as shown in the lower part of Figure 14, the comparison method incorrectly detects 2D-BBOXes representing people other than the intended target, the player, but this incorrect detection is eliminated in this method.

[0053] Furthermore, the information processing device according to the above embodiment can be applied, for example, to a scoring system for gymnastics competitions. Here, with reference to Figure 15, an outline of the processing of a gymnastics scoring system will be described.

[0054] When a multi-view image is input to the scoring system, it detects the region of a person from each image included in the multi-view image. Next, the scoring system determines whether the person indicated by the detected region is an athlete or not, based on whether the location of the person is within the competition area, and identifies the region indicating an athlete. The scoring system tracks the athlete by associating the regions indicating the same athlete with the multi-view images in a time series. From each of the tracked images, the scoring system recognizes the athlete's 2D skeletal information using a recognition model, etc. From the 2D skeletal information, the scoring system estimates 3D skeletal information using camera parameters. Then, the scoring system performs post-processing such as smoothing on the 3D skeletal information in the time series, estimates the phases (breaks) of the performance, and then recognizes the technique.

[0055] In the scoring system described above, a detector that has been machine-learned using pseudo-labels generated by the information processing device according to the above embodiment can be applied to the process of detecting the region of a person.

[0056] In the above embodiment, the case in which the estimated 3D position information, which is 3D orientation information, is projected onto all images included in the multi-view image was described, but the embodiment is not limited to this. The projection may be applied to at least one image in the multi-view image, such as targeting images in which the 2D-BBOX has not been detected by the detector.

[0057] Furthermore, the disclosure technology is not limited to gymnasts; it can be applied to various individuals, including athletes from other sports, and ordinary pedestrians. In addition, it can be applied to objects other than people, such as animals and vehicles.

[0058] Furthermore, in the above embodiment, the information processing program is pre-stored (installed) in the storage device, but the invention is not limited thereto. The program relating to the disclosed technology may be provided in a form stored on a storage medium such as a CD-ROM, DVD-ROM, or USB memory. [Explanation of symbols]

[0059] 10 Information Processing Devices 11 Acquisition Department 12 Estimation part 13 Generation part 14 Selection Section 15 Machine Learning Department 22 detectors 24 Camera Parameter DB 30n camera 40n image 42n 2D-BBOX 44n pseudo-label 50 Computers 51 CPU 52 memory 53 Storage device 54 Input / Output Devices 55 R / W device 56 Communication I / F 57 Bus 59 Storage medium 60 Information Processing Programs 61 Acquisition process control instruction 62 Estimated Process Control Instructions 63 Generation Process Control Instructions 64 Select Process Control Instructions 65 Machine Learning Process Control Instructions 70 Information storage area 90 Object

Claims

1. By acquiring multiple images taken by each of multiple cameras that photograph the object from multiple different viewpoints, Based on the two-dimensional position information of the object detected from each of the multiple images and the camera parameters of each of the multiple cameras, the three-dimensional position information of the object is estimated. Based on the camera parameters of the camera that captured at least one of the aforementioned plurality of images, the three-dimensional position information of the object is projected onto the at least one image, and the two-dimensional position information of the object in the at least one image is calculated. The calculated two-dimensional position information of the object is assigned as a pseudo-label to at least one image to generate training data for performing machine learning on a machine learning model that detects the two-dimensional position information of the object from the image. An information processing method in which a computer performs a process that includes the following.

2. The information processing method according to claim 1, wherein the process for calculating the two-dimensional position information of the object includes calculating information indicating the region of the object.

3. If the object is a person, the process for estimating the three-dimensional position information of the object includes detecting the two-dimensional posture information of the object from each of the plurality of images as the two-dimensional position information of the object, and estimating the three-dimensional posture information of the object based on the two-dimensional posture information of the object, according to claim 1 or claim 2.

4. The information processing method according to claim 1 or 2, wherein the computer performs a process that includes performing machine learning on the machine learning model using the training data.

5. The information processing method according to claim 1 or claim 2, wherein the training data is an image to which the pseudo-labels are assigned, wherein the three-dimensional position information of the object from which the projection is made is included in a predetermined range in three-dimensional space.

6. The information processing method according to claim 5, wherein, in the case of the object, the predetermined range is the competition area corresponding to the sport.

7. The information processing method according to claim 1 or 2, wherein the images to which the pseudo-labels are assigned, wherein the time of capture of the corresponding image falls within a predetermined time range, are used as the training data.

8. The information processing method according to claim 7, wherein, in the case of the object, the predetermined time range is a time range corresponding to the period from the start to the end of the performance.

9. The information processing method according to claim 4, which repeatedly performs the generation of pseudo-labels and the machine learning of the machine learning model that uses the images to which the pseudo-labels have been assigned as training data.

10. The information processing method according to claim 9, wherein the degree of overlap between the region indicated by the two-dimensional position information of the detected object and the region indicated by the generated pseudo-label is less than a predetermined threshold, and the image to which the pseudo-label is assigned is excluded from the training data.

11. An acquisition unit that acquires multiple images taken by each of multiple cameras that photograph an object from multiple different viewpoints, An estimation unit estimates the three-dimensional position information of the object based on the two-dimensional position information of the object detected from each of the plurality of images and the camera parameters of each of the plurality of cameras. A calculation unit that projects the three-dimensional position information of the object onto the at least one image based on the camera parameters of the camera that captured at least one of the plurality of images, and calculates the two-dimensional position information of the object in the at least one image. A generation unit that assigns the calculated two-dimensional position information of the object as a pseudo-label to at least one image and generates training data for performing machine learning on a machine learning model that detects the two-dimensional position information of the object from the image, Information processing device including

12. By acquiring multiple images taken by each of multiple cameras that photograph the object from multiple different viewpoints, Based on the two-dimensional position information of the object detected from each of the multiple images and the camera parameters of each of the multiple cameras, the three-dimensional position information of the object is estimated. Based on the camera parameters of the camera that captured at least one of the aforementioned plurality of images, the three-dimensional position information of the object is projected onto the at least one image, and the two-dimensional position information of the object in the at least one image is calculated. The calculated two-dimensional position information of the object is assigned as a pseudo-label to at least one image to generate training data for performing machine learning on a machine learning model that detects the two-dimensional position information of the object from the image. An information processing program that causes a computer to perform a process that includes the following.

Images