MODEL GENERATION METHOD, MODEL GENERATION DEVICE, MODEL GENERATION PROGRAM, MOBILE BODY ATTITUDE ESTIMATION METHOD, AND MOBILE BODY ATTITUDE ESTIMATION DEVICE

Abstract

To provide a model production method, a model production device, a model production program, a moving body posture estimation method, and a moving body posture estimation device which can further easily create a machine learning model for detecting a moving body.SOLUTION: A model production method includes: teacher data creation step S101 of indicating at least one of image coordinates, in a 2D image including at least a moving body captured, of vertices of a rectangular shape formed by projecting a contour of the moving body when viewed from above on a road, as a key point of the moving body and creating the 2D image with information on the key point added thereto as teacher data; and model production step S102 of performing machine learning using the teacher data to produce a corner model which outputs the key point from the 2D image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a model generation method, a model generation device, a model generation program, a moving body attitude estimation method, and a moving body attitude estimation device. [Background technology]

[0002] In recent years, various technologies for detecting objects such as vehicles have been developed. For example, Non-Patent Document 1 describes a method for creating training data by collating two-dimensional images acquired by a monocular camera with distance information (point cloud data) acquired by a LiDAR, which is three-dimensional data, to create a machine learning model for detecting vehicles. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Garrick Brazil, et al., "M3D-RPN: Monocular 3D Region Proposal Network for Object Detection", Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), 2019, p.9287-9296 Summary of the Invention [Problem to be solved by the invention]

[0004] In Non-Patent Document 1, since training data is created by comparing two-dimensional data with three-dimensional data, it may take a huge amount of time to collect sufficient training data. In addition, since three-dimensional data is used, the calculation time required to create a machine learning model may also be enormous.

[0005] The present invention has been made to solve such problems, and aims to provide a model generation method, a model generation device, a model generation program, a moving body posture estimation method, and a moving body posture estimation device that can more easily create a machine learning model for moving body detection. [Means for solving the problem]

[0006] The model generation method of the present invention includes a teacher data creation step of specifying, as a key point of the moving body, the image coordinates of at least one of the vertices of a rectangular shape formed by projecting the outline of the moving body when viewed from above onto a road, in a two-dimensional image of the moving body, and creating, as teacher data, the two-dimensional image to which information about the key point has been added; and a model generation step of performing machine learning using the teacher data to generate a machine learning model that outputs the key points from the two-dimensional image.

[0007] The model generation device of the present invention includes a teacher data creation unit that specifies, as a key point of the moving body, the image coordinates of at least one of the vertices of a rectangular shape formed by projecting the outline of the moving body when viewed from above onto a road, in a two-dimensional image of at least a moving body, and creates, as teacher data, the two-dimensional image to which information about the key point is added, and a learning unit that performs machine learning using the teacher data and generates a machine learning model that outputs the key points from the two-dimensional image.

[0008] The model generation program of the present invention causes a model generation device to execute a teacher data creation process in which image coordinates in a two-dimensional image of at least a moving body, which are the image coordinates of at least one of the vertices of a rectangular shape formed by projecting the outline of the moving body when viewed from above onto a road, are specified as key points of the moving body, and the two-dimensional image to which information about the key points is added is created as teacher data, and a model generation process in which machine learning is performed using the teacher data to generate a machine learning model that outputs the key points from the two-dimensional image.

[0009] The moving body posture estimation method of the present invention comprises an inference step of inferring the key points of the moving body from the two-dimensional image using a machine learning model generated by the above-mentioned model generation method, and an output step of performing calculations necessary to estimate the posture of the moving body for the key points.

[0010] The moving body posture estimation device of the present invention includes an inference unit that infers the key points of the moving body from the two-dimensional image using a machine learning model generated by the above-mentioned model generation method, and an output unit that performs the necessary calculations to estimate the posture of the moving body for the key points. Effect of the Invention

[0011] According to the model generation method, model generation device, model generation program, moving body posture estimation method, and moving body posture estimation device of the present invention, teacher data can be created using only two-dimensional images of at least a moving body. In other words, teacher data can be created without using three-dimensional data such as LiDAR. Therefore, the time required to collect teacher data can be shortened compared to when three-dimensional data is used. In addition, since three-dimensional data is not used, the calculation time required to create a machine learning model can also be shortened. Therefore, it is possible to provide a model generation method, model generation device, model generation program, moving body posture estimation method, and moving body posture estimation device that can more easily create a machine learning model for detecting a moving body. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of a configuration of a moving object posture estimation device according to a first embodiment of the present invention. [Diagram 2] 3 is a flowchart showing a moving object posture estimation method according to the first embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram illustrating a bounding box and key points according to the first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing a neural network structure of a corner model in accordance with the first embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram for explaining an operation using VGG16, which is an example of a Base Net according to the first embodiment of the present invention. [Figure 6] FIG. 2 is a diagram for explaining an example of a calculation using a Spatial Net according to the first embodiment of the present invention. [Figure 7] FIG. 2 is a diagram illustrating an anchor according to the first embodiment of the present invention. [Figure 8] 4 is a flowchart showing a process for creating teacher data according to the first embodiment of the present invention. [Figure 9] 5 is a flowchart showing a process for generating a corner model according to the first embodiment of the present invention. [Figure 10] 4 is a flowchart showing a moving object posture estimation process according to the first embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] First embodiment Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following first embodiment. In addition, the following description and drawings are appropriately simplified for clarity of explanation.

[0014] Fig. 1 is a block diagram showing an example of the configuration of a moving object posture estimation apparatus 100 according to the embodiment 1. As shown in Fig. 1, the moving object posture estimation apparatus 100 includes a model generation block 110 and an inference block 120. The model generation block 110 functions as a model generation device. The model generation block 110 also includes an image acquisition device 111, an input unit 112, a teacher data creation unit 113, a learning unit 114, a model storage unit 130, and the like. The inference block 120 also includes an image acquisition device 121, an input unit 122, an inference unit 123, an output unit 124, a model storage unit 130, and the like. In other words, the model generation block 110 and the inference block 120 share the model storage unit 130 . Then, the moving body posture estimation device 100 uses a corner model (machine learning model) previously generated by machine learning by the model generation block 110 to output key points of the moving body required to estimate the posture of the moving body from a two-dimensional image acquired by the image acquisition device 121.

[0015] FIG. 2 shows a flowchart illustrating a moving object posture estimation method according to the first embodiment. As shown in FIG. 2, first, the model generation block 110 creates training data (step S101; training data creation step). Next, the model generation block 110 performs machine learning using the training data to generate a corner model that outputs key points of the moving object required to estimate the posture of the moving object from a two-dimensional image (step S102; model generation step). Next, the inference block 120 uses the corner model to infer key points of the moving object from a two-dimensional image acquired by the image acquisition device 121 (step S103; inference step). Next, the inference block 120 performs a predetermined calculation process on the key points (image coordinates) and outputs the key points that have been calculated (step S104; output step). The model generation block 110 and the inference block 120 are described in more detail below.

[0016] <Configuration of model generation block 110> The image capture device 111 is a camera that captures a two-dimensional image of a scene including at least a moving object, such as a traffic flow monitoring camera that captures an image of a road. The image capture device 111 inputs a plurality of two-dimensional images to the input unit 112.

[0017] The input unit 112 performs necessary processing on the two-dimensional image input from the image acquisition device 111 to make the image data suitable for processing in the teacher data creation unit 113 and the learning unit 114. For example, the input unit 112 performs processing to adjust the data size by adjusting the resolution of the two-dimensional image, processing to make it easier to extract feature amounts such as edge emphasis, processing to determine an area where processing is not performed, etc. The input unit 112 inputs the two-dimensional image that has been processed to the teacher data creation unit 113 and the learning unit 114.

[0018] The teacher data creation unit 113 creates teacher data by adding information about key points of the moving object, which is necessary for estimating the posture of the moving object, to the two-dimensional image input from the input unit 112 .

[0019] Specifically, the teacher data creation unit 113 creates a general bounding box on the two-dimensional image. Here, the bounding box is a rectangular frame surrounding the area of ​​an object (object area) captured in the two-dimensional image. FIG. 3 shows a bounding box BX and key points P1, P2, P3, and P4 created by the teacher data creation unit 113. In the example shown in FIG. 3, there is an image area of ​​a truck photographed from the left diagonal rear on the two-dimensional image, and a bounding box BX (shown by a dashed line in FIG. 3) surrounding the image area of ​​the truck is created. The bounding box may be created by a known technique for creating a bounding box, such as an object recognition technique for recognizing an object on a two-dimensional image. The bounding box may also be created manually by an operator who operates the moving body posture estimation device 100. Furthermore, the bounding box created by the teacher data creation unit 113 by executing a known object recognition algorithm may be corrected by the operator.

[0020] Next, the teacher data creation unit 113 assigns to each object in each bounding box a classification (class) tag (hereinafter referred to as a "classification tag") indicating which classification (e.g., passenger car, truck, pedestrian, bicycle, motorcycle, etc.) the object belongs to. The classification tag may be assigned manually by an operator who operates the moving body posture estimation device 100, or may be assigned by the teacher data creation unit 113 executing an algorithm for assigning a known classification tag. The classification tag may also include more detailed information. For example, the classification tag may include a ratio (subjective numerical value) of the imaged portion to the entire object when only a portion of the object is imaged on a two-dimensional image. The classification tag may also include a ratio (subjective numerical value) of the portion of one object hidden by another object to the entire object when objects are imaged overlapping each other on a two-dimensional image.

[0021] Next, the teacher data creation unit 113 designates key points of objects that are given classification tags indicating moving objects, such as pedestrians, cars, trucks, bicycles, and motorcycles, among the objects enclosed by the bounding box. Specifically, the teacher data creation unit 113 designates image coordinates of vertices of a rectangular shape formed by projecting the outer shape of an object given a classification tag indicating a moving object (hereinafter simply referred to as a "moving object") onto a road when the object is viewed from above, as key points of the moving object. Here, the image coordinates are coordinates for indicating the position of a pixel on a two-dimensional image, and are, for example, coordinates that are defined as having the center of the pixel located at the leftmost and uppermost side of the two-dimensional image as the origin, the left-right direction or horizontal direction as the x direction, and the up-down direction or vertical direction as the y direction. In addition, when the moving object is viewed from above, the outer shape of the moving object is not strictly limited to a rectangular shape, but for example, the substantially smallest rectangular shape including the outer shape of the moving object when viewed from above is set as the rectangular shape when designating the key points. The teacher data creation unit 113 may create the rectangular shape using a deep learning network that the teacher data creation unit 113 has created by performing machine learning in advance. The rectangular shape may be created manually by an operator who operates the moving body posture estimation device 100. The operator may also modify the rectangular shape created by the teacher data creation unit 113. For moving bodies other than pedestrians, bicycles, and motorcycles, the teacher data creation unit 113 may specify the image coordinates of the four vertices of the rectangular shape as four key points. For bicycles and motorcycles among the moving bodies, the teacher data creation unit 113 may specify the image coordinates of two vertices of the rectangular shape, one front point and one rear point corresponding to the one front point, as key points. In other words, for bicycles and motorcycles among the moving bodies, only two key points are specified. Furthermore, with regard to pedestrians among the moving objects, the teacher data creation unit 113 may specify as a key point one center point of the approximately circular shape obtained by projecting the pedestrian's head onto the road when viewed from above, or may specify two key points one center point of the pedestrian's toes projected onto the road when viewed from above.Furthermore, the teacher data creation unit 113 may designate, as a key point, one of the vertices of the minimum rectangular shape that substantially includes the outline of the pedestrian when the pedestrian is viewed from above. Regardless of the type of moving object, the teacher data creation unit 113 may designate as a key point only one of the four vertices of a rectangular shape formed by projecting the outer shape of the moving object onto a road. In the example shown in FIG. 3, a rectangular shape (shown by a thick solid line in FIG. 3) is formed on the road, and the image coordinates of the vertices of the rectangular shape are specified as key points P1, P2, P3, and P4. As a result, the image coordinates of pixels on the road in a two-dimensional image are specified as key points, so that the area of ​​a moving object on the road surface can be calculated from the rectangular shape defined by the four key points, and the size of the moving object can be estimated. Specifically, the physical coordinates (ortho coordinates) of landmark points on the road are calibrated with the image coordinates to obtain a transformation matrix (homography matrix), and the image coordinates of the four key points are converted into ortho coordinates, making it possible to estimate the size of the moving object. Note that ortho coordinates are image coordinates in an ortho image.

[0022] Furthermore, when the teacher data creation unit 113 designates key points, the key points are designated in a predetermined order. In the example shown in FIG. 3, the key points are designated in the order of the left front P1, the right front P2, the right rear P3, and the left rear P4 of the moving body. Here, since the truck shown in FIG. 3 is photographed from the left diagonal rear, the right front key point P2 is the estimated position. This makes it possible to estimate the direction (posture) of the moving body from information regarding the designation order of the key points. Note that, when the moving body is a bicycle or a motorbike, the vertices of a rectangular shape formed by projecting the outline of the bicycle or motorbike onto the road may be designated in the order of one front point and one rear point corresponding to the one front point. This makes it possible to estimate the direction (posture) of the bicycle or motorbike from information regarding the designation order of the key points.

[0023] Furthermore, when the moving body is a vehicle (excluding bicycles and motorcycles), the teacher data creation unit 113 designates the key points so that the ground contact positions of the wheels located on at least one of the right and left sides of the moving body are aligned with a line connecting two key points located in front and behind on at least one of the right and left sides of the moving body. This makes it possible to estimate the position of the moving body based on the position information of the imaging range of the image acquisition device 121 and the image coordinates of the key points in the captured two-dimensional image. Specifically, the position of the moving body can be estimated by calibrating the physical coordinates (ortho coordinates) of the landmark points on the road and the image coordinates to obtain a transformation matrix (homography matrix), and converting the image coordinates of the four key points into ortho coordinates. In addition, it becomes possible to estimate the traveling direction of the moving body from the orientation of the moving body and the time change in the position of the moving body.

[0024] Furthermore, the teacher data creation unit 113 inputs, as teacher data, to the learning unit 114, a two-dimensional image to which information regarding the image coordinates of the vertices of the bounding box, the classification tag, and the key points (image coordinates) has been added.

[0025] The learning unit 114 performs machine learning using the teacher data input from the teacher data creation unit 113 and the corner model before or during learning, and generates a trained corner model. Here, the corner model is a machine learning model that outputs key points from a two-dimensional image. FIG. 4 shows the neural network structure of the corner model. As shown in FIG. 4, the corner model has a neural network structure including a base net for extracting feature amounts from a two-dimensional image, a spatial net for performing a multi-resolution analysis on the feature amounts to create a multi-resolution feature map, and a classifier for outputting the key points based on the multi-resolution feature map.

[0026] First, the learning unit 114 extracts feature amounts from the two-dimensional image input from the input unit 112, using the Base Net shown in Fig. 4. Here, the learning unit 114 extracts a plurality of feature amounts from the two-dimensional image. As the base net, neural networks such as VGG16, ResNet, and EffecientNet can be used. Here, an example will be described in which VGG16 is used as the Base Net. Specifically, the learning unit 114 repeatedly performs convolution and pooling operations on the two-dimensional image input from the input unit 112 using VGG16 to extract features. Here, the convolution operation is an operation of applying a local filter to the two-dimensional image, and the pooling operation is an operation of reducing the image size. FIG. 5 shows an example of an operation using VGG16. In FIG. 5, Conv1-1, Conv1-2, Conv2-1, Conv2-2, Conv3-1, ... indicate convolution operations, and Max pooling indicates pooling operations. And, for example, Conv4-3(64,64,516)(3*3) means an operation of generating a 516-channel image with a 3*3 local filter for an image size of 64*64. After performing the calculation, the learning unit 114 applies a ReLU function (Recified Linear Unit) as an activation function to each pixel value of the generated image to generate a feature map as a feature amount. For example, in FIG. 5, ReLU_4_3 means a calculation to apply a ReLU function to a 516-channel image generated by the calculation of Conv4-3 (64,64,516) (3*3) to generate a feature map. Similarly, in FIG. 5, ReLU_7 means a calculation to apply a ReLU function to a 1032-channel image generated by the calculation of Conv7-1 (32,32,1032) (3*3) to generate a feature map. Here, the ReLU function is, for example, a piecewise linear function represented by the following formula (1).

number

[0027] Next, the learning unit 114 performs a multi-resolution analysis using the Spatial Net shown in FIG. 4, and creates a multi-resolution feature map based on a plurality of feature amounts extracted from the two-dimensional image. Specifically, the learning unit 114 distributes the extracted plurality of feature amounts (feature map) for each scale in the multi-resolution analysis, and creates a multi-resolution feature map. FIG. 6 shows an example of a calculation using the Spatial Net. As shown in FIG. 6, the Spatial Net is a neural network that repeatedly performs convolution calculations (Conv8-1, Conv8-2, Conv8-3, Conv8-4) to generate a multi-resolution feature map for dealing with changes in image size. Here, the "multi-resolution feature map" means a plurality of feature maps with different resolutions that are generated by gradually reducing the resolution of the feature map. Each feature map included in the multi-resolution feature map is also called a "layer." Furthermore, "scale" means the resolution of each feature map. Furthermore, "distributing a feature map for each scale" means gradually reducing the resolution by half and decomposing the feature map for each resolution. For example, in FIG. 6, ReLU_8 means an operation to generate a feature map with a resolution reduced by two levels by applying the ReLU function to the 512-channel image generated by the operations of Conv8-1(32,32,256)(1*1) and Conv8-2(16,16,512)(3*3) on ReLU_4_3(64,64,512), which is the output of the Base Net. Similarly, in Fig. 6, ReLU_9 means an operation to generate a feature map with a resolution reduced by 4 steps by applying a ReLU function to a 512-channel image generated by the operations of Conv8-1(32,32,256)(1*1), Conv8-2(16,16,512)(3*3), Conv8-3(16,16,256)(1*1), and Conv8-4(8,8,512)(3*3) on ReLU_4_3(64,64,512), which is the output of the Base Net, to generate a feature map with the resolution reduced by 4 steps. Also, in Fig. 6, ReLU_7 means that a convolution operation is not performed on ReLU_4_3(64,64,512), which is the output of the Base Net, that is, a feature map with the same resolution is generated.In addition, in Fig. 6, ReLU_4_3_norm means that a feature map is generated by performing normalization processing (I2normalize) on ReLU_4_3 (64,64,512), which is the output of the Base Net. Here, I2normalize is a process of dividing a multidimensional vector x by its Euclidean norm (the square root of the sum of the squares of each component: magnitude) and normalizing the magnitude to 1.

[0028] Next, the learning unit 114 uses a classifier to perform classification processing based on the multi-resolution feature map, and outputs key points. Specifically, as shown in FIG. 4, the learning unit 114 assigns classification tags and performs bounding box (frame) regression based on the multi-resolution feature map, and outputs image coordinates of key points (corners). More specifically, the learning unit 114 prepares anchors for each of multiple layers (multiple feature maps with different resolutions) of the multi-resolution feature map. Here, an "anchor" is a template frame according to the shape of the recognition target, and is a bounding box candidate. The number of "anchors" set varies depending on the layer. In the example shown in FIG. 7, three rectangular anchors A1, A2, and A3 are set for a certain layer (feature map with a certain resolution). For example, the learning unit 114 sets anchors for each of multiple layers with different resolutions (ReLU_4_3_norm, ReLU_7, ReLU_8, and ReLU_9) included in the multi-resolution feature map output from the Spatial Net. Next, the learning unit 114 assigns a classification tag to each anchor, performs bounding box regression, and detects key points. Then, the learning unit 114 outputs the classification tag, the bounding box regression calculation result, and the image coordinates of the key points (corners) for each anchor.

[0029] Next, the learning unit 114 compares the result (output value) output by performing classification processing based on the multi-resolution feature map using a classifier with the teacher data (correct value) to calculate a residual (difference). For example, the learning unit 114 calculates the residual by the following formula (2).

number

number

number

number

number

number

number

number

number

[0030] Next, the learning unit 114 judges whether the residual is smaller than a predetermined threshold. If the residual is equal to or larger than the predetermined threshold, the learning unit 114 updates the weights in the Base Net, the Spatial Net, and the classifier, and performs the above-mentioned process again. If the residual is smaller than the predetermined threshold, the learning unit 114 judges that the machine learning is completed, and stores the corner model with the updated weights in the model storage unit 130. Note that the predetermined threshold is a relatively small value determined according to the purpose, and is, for example, a value smaller than 1. Alternatively, the learning unit 114 may end the calculation after performing the process of steps S301 to 305 shown in FIG. 9 described later a predetermined number of times, even if the residual is equal to or larger than the predetermined threshold. As a result of the above, a machine-learned corner model is generated.

[0031] <Configuration of inference block 120> The image capture device 121 is a camera that captures a two-dimensional image of a scene including at least a moving object, such as a traffic flow monitoring camera that captures an image of a road, similar to the image capture device 111. The image capture device 121 inputs a plurality of two-dimensional images to the input unit 122.

[0032] The input unit 122 performs necessary processing on the two-dimensional image input from the image acquisition device 121 to make the image data suitable for processing in the inference unit 123. For example, the input unit 122 performs processing to adjust the data size by adjusting the resolution of the two-dimensional image, processing to make it easier to extract feature amounts such as edge emphasis, processing to determine a range where processing is not performed, etc. The input unit 122 inputs the two-dimensional image that has been processed to the inference unit 123.

[0033] The inference unit 123 infers key points from the two-dimensional image input from the input unit 122, using the corner model stored in the model storage unit 130. The inference unit 123 also inputs the inferred key points to the output unit 124.

[0034] Specifically, first, the inference unit 123 uses the base net of the corner model to extract feature amounts from the two-dimensional image input from the input unit 122. Here, the inference unit 123 extracts a plurality of feature amounts from the two-dimensional image. The process of extracting a plurality of feature amounts from the two-dimensional image by the inference unit 123 is similar to the process of extracting a plurality of feature amounts from the two-dimensional image by the learning unit 114, and therefore a detailed description thereof will be omitted.

[0035] Next, the inference unit 123 performs a multi-resolution analysis of the corner model using a spatial net, distributes a plurality of feature amounts extracted from the two-dimensional image for each scale in the multi-resolution analysis, and creates a multi-resolution feature map. The process of creating the multi-resolution feature map by the inference unit 123 is similar to the process of creating the multi-resolution feature map by the learning unit 114, and therefore a detailed description thereof will be omitted.

[0036] Next, the inference unit 123 uses a classifier to perform classification processing based on the multi-resolution feature map, and outputs key points. Specifically, the inference unit 123 assigns classification tags and performs bounding box (frame) regression based on the multi-resolution feature map, and outputs image coordinates of key points (corners). The process of outputting key points by the inference unit 123 is similar to the process of outputting key points by the learning unit 114, and therefore a detailed description will be omitted.

[0037] The output unit 124 performs a calculation required to estimate the posture of the moving body for the key points input from the inference unit 123. Specifically, the output unit 124 converts the key points (image coordinates) into values ​​required to calculate the size, orientation (posture) of the moving body, position, and traveling direction of the moving body from the key points. Specifically, the output unit 124 calibrates the physical coordinates (ortho coordinates) of the landmark points on the road with the image coordinates to obtain a transformation matrix (homography matrix), and converts the image coordinates of the four key points into ortho coordinates using the transformation matrix. This makes it possible to calculate the size, orientation (posture) of the moving body, position, and traveling direction of the moving body from the ortho coordinates of the four key points.

[0038] <Model generation method> Next, a model generation method according to the first embodiment will be described with reference to Fig. 8 and Fig. 9. Fig. 8 shows the teacher data creation process, which is the process of step S101 shown in Fig. 2. Fig. 9 shows the corner model generation process, which is the process of step S102 shown in Fig. 2.

[0039] First, as shown in FIG. 8, the input unit 112 inputs the two-dimensional image acquired by the image acquisition device 111 to the teacher data creation unit 113 (step S201).

[0040] Next, the teacher data creation unit 113 creates a bounding box on the two-dimensional image (step S202; detection process). When multiple objects are included in the two-dimensional image, the teacher data creation unit 113 creates multiple bounding boxes on the two-dimensional image.

[0041] Next, the teacher data creation unit 113 assigns a classification tag to each object in the bounding box (step S203; classification process).

[0042] Next, the teacher data creation unit 113 designates key points (image coordinates) of an object to which a classification tag indicating a moving object has been added (step S204; extended detection process). In this specification, the "detection process" in step S202 refers to creating a bounding box, and the "extended detection process" in step S204 refers to designating key points in addition to the bounding box.

[0043] Next, as shown in FIG. 9, teacher data is input from teacher data creation unit 113 to learning unit 114 (step S301).

[0044] Next, the learning unit 114 uses the corner model being learned to output key points of the moving object (step S302).

[0045] Next, the learning unit 114 compares the output value output in step S302 with teacher data (correct value) to calculate a residual (step S303).

[0046] Next, the learning unit 114 determines whether the residual calculated in step S303 is smaller than a predetermined threshold value (step S304).

[0047] In step S304, if the residual is equal to or greater than a predetermined threshold (step S304; No), the learning unit 114 updates the weights in the Base Net, the Spatial Net, and the classifier (step S305), and returns to the process of step S301.

[0048] In step S304, if the residual is smaller than the predetermined threshold (step S304; Yes), the learning unit 114 determines that the machine learning is completed, stores the learned corner model in the model storage unit 130, and ends this process.

[0049] <Mobile posture estimation method> Next, the moving object posture estimation method according to the first embodiment will be described with reference to Fig. 10. Fig. 10 shows the process of creating teacher data, which is the process of steps S103 and S104 shown in Fig. 2.

[0050] First, as shown in FIG. 10, the input unit 122 performs a predetermined process on the two-dimensional image input from the image acquisition device 121, and inputs the result to the inference unit 123 (step S401).

[0051] Next, the inference unit 123 extracts features from the two-dimensional image input in step S401, using the Base Net of the trained corner model stored in the model storage unit 130 (step S402; feature extraction process).

[0052] Next, the inference unit 123 creates a multi-resolution feature map using the spatial net of the corner model (step S403; distribution process).

[0053] Next, the inference unit 123 uses a classifier of the corner model to assign classification tags and perform bounding box (frame) regression based on the multi-resolution feature map, and output key point (corner) image coordinates (step S404; classification process).

[0054] Next, the output unit 124 performs calculations required for estimating the posture of the moving object and the like on the key points (output values) input from the inference unit 123, and ends this process (step S405).

[0055] According to the model generation method, the model generation block 110 as a model generation device, the model generation program, the moving body posture estimation device 100, and the moving body posture estimation method according to the first embodiment described above, teacher data can be created using only two-dimensional images of at least a moving body. In other words, teacher data can be created without using three-dimensional data such as LiDAR. Therefore, compared to the case where three-dimensional data is used, the time required to collect teacher data can be shortened. In addition, since three-dimensional data is not used, the calculation time required to create a machine learning model can also be shortened. Therefore, it is possible to provide a model generation method, a model generation device, a model generation program, a moving body posture estimation method, and a moving body posture estimation device that can more easily create a machine learning model for detecting a moving body.

[0056] Furthermore, from at least a two-dimensional image capturing a moving object, image coordinates in the two-dimensional image, which are image coordinates of vertices of a rectangular shape formed by projecting the outer shape of the moving object when viewed from above onto a road, are specified as four key points of the moving object. Therefore, since image coordinates of pixels on the road in the two-dimensional image are specified as key points, the area of ​​the moving object on the road surface can be calculated from the rectangular shape defined by the four key points, and the size of the moving object can be estimated. Therefore, even if the training data is created using only two-dimensional images, a machine learning model capable of detecting moving objects with the accuracy required for road control, etc. can be generated.

[0057] Furthermore, when specifying key points, the teacher data creation unit 113 specifies the key points in a predetermined order, which makes it possible to estimate the orientation (posture) of the moving object from the information on the key points.

[0058] Furthermore, when the moving body is a vehicle (excluding bicycles and motorcycles), the teacher data creation unit 113 specifies the key points so that the ground contact positions of the wheels located on at least one of the right and left sides of the vehicle are aligned with a line connecting two key points located at the front and rear on at least one of the right and left sides of the vehicle. This makes it possible to estimate the position of the vehicle based on the position information of the imaging range of the image acquisition device 121 and the image coordinates of the key points in the captured two-dimensional image. It also makes it possible to estimate the traveling direction of the vehicle from the orientation of the vehicle and the change over time in the position of the vehicle.

[0059] In addition, the corner model (machine learning model) has a neural network structure including a base net for extracting features from a two-dimensional image, a spatial net for performing multi-resolution analysis on the features to create a multi-resolution feature map, and a classifier for outputting key points based on the multi-resolution feature map. Therefore, since the corner model has a relatively simple neural network structure, it is possible to estimate a moving object or the posture of the moving object at a speed required for traffic control, etc.

[0060] The present invention is not limited to the above-mentioned embodiment, and can be modified as appropriate without departing from the spirit of the present invention. For example, in the above-mentioned embodiment, the teacher data creation unit 113 may specify feature points (end points, center points, etc.) of a three-dimensional object as key points. In this case, it is possible to estimate the position and posture of the object. In addition, in the above-mentioned embodiment, when the moving object is a vehicle (bicycle, motorcycle), the teacher data creation unit 113 specifies the image coordinates of four vertices of a rectangular shape as four key points, and when the moving object is a bicycle or motorcycle, the teacher data creation unit 113 specifies the image coordinates of two vertices, one at the front and one at the rear, of the rectangular shape as two key points, but the teacher data creation unit 113 may specify only one arbitrary point of the four vertices of the rectangular shape as one key point. Even in this case, it is possible to estimate the posture of the vehicle by combining road information obtained by a traffic flow monitoring camera with the one key point. In other words, if one key point is output for a moving object by the machine learning model (corner model) generated by the model generating device according to this embodiment, it becomes possible to estimate the posture of the moving object using road information obtained by a traffic flow monitoring camera. In this case, the residual of the corner point (key point) can be calculated using the above formula (7). In addition, the teacher data creating unit 113 may specify five or more key points including the four vertices of a rectangular shape, or five or more key points unrelated to the vertices of a rectangular shape, for the moving object. Furthermore, the number of key points specified by the teacher data creating unit 113 is not limited to the number shown in the above embodiment. When the teacher data creating unit 113 specifies m key points (m is a positive integer), the residual of the m corner points (key points) is calculated by the following formula (10).

number

[0061] In the above embodiment, the present invention has been described as a hardware configuration, but the present invention is not limited to this. The present invention can also be realized by having a CPU (Central Processing Unit) execute a computer program to perform the processing procedures shown in the flowcharts of Figures 2, 8 to 10.

[0062] The above-mentioned program may be stored using a ROM and provided to a computer. The above-mentioned program may be stored using various types of non-transitory computer readable media and provided to a computer. The non-transitory computer readable media includes various types of tangible storage media. Examples of the non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may be provided to a computer by various types of transitory computer readable media. Examples of the transitory computer readable media include electric signals, optical signals, and electromagnetic waves. The transitory computer readable media can provide the program to a computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path. [Explanation of symbols]

[0063] 100 Mobile body pose estimation device 110 Model generation block (model generation device) 111 Image Acquisition Device 112 Input section 113 Teacher Data Creation Department 114 Learning Department 120 Inference Blocks 121 Image Acquisition Device 122 Input section 123 Reasoning part 124 Output section 130 Model Storage Unit P1, P2, P3, P4 Key Points BX Bounding Box

Claims

1. a teacher data creation step of designating, as a key point of the moving body, image coordinates in a two-dimensional image of at least a moving body, the image coordinates of at least one of the vertices of a rectangular shape formed by projecting the outline of the moving body when viewed from above onto a road, and creating, as teacher data, the two-dimensional image to which information about the key point is added; a model generation step of performing machine learning using the training data to generate a machine learning model that outputs the key points from the two-dimensional image; Equipped with In the teacher data creation step, when image coordinates of two or more vertices of the rectangular shape are designated as the key points of the moving object, A model generation method, wherein in the teacher data creation step, the key points in the two-dimensional image are indicated in a predetermined order.

2. 2. The model generating method according to claim 1, wherein in said teacher data generating step, four vertices of said rectangular shape are specified as said four key points of said moving object.

3. the moving object is a vehicle, 3. The model generation method according to claim 1, wherein in the teacher data creation step, the key points are instructed so that the ground contact positions of wheels located on at least one of the right and left sides of the vehicle are aligned along a line connecting two of the key points located at the front and rear on at least one of the right and left sides of the vehicle.

4. 4. The model generating method according to claim 1, wherein the machine learning model generated in the model generating step has a neural network structure including: a Base Net for extracting features from the two-dimensional image; a Spatial Net for performing a multi-resolution analysis on the features to create a multi-resolution feature map; and a classifier for outputting the key points based on the multi-resolution feature map.

5. a teacher data creation unit that designates, as a key point of the moving body, image coordinates of at least one of the vertices of a rectangular shape formed by projecting the outline of the moving body when viewed from above onto a road, in a two-dimensional image of at least a moving body, and creates, as teacher data, the two-dimensional image to which information about the key point is added; a learning unit that performs machine learning using the training data and generates a machine learning model that outputs the key points from the two-dimensional image; Equipped with A model generating device in which, when the image coordinates of two or more of the vertices of the rectangular shape are designated as the key points of the moving body, the teacher data creation unit designates the key points in the two-dimensional image in a predetermined order.

6. The model generating device includes: a training data creation process for designating, as a key point of the moving body, image coordinates of at least one of the vertices of a rectangular shape formed by projecting the outline of the moving body when viewed from above onto a road, in a two-dimensional image of at least a moving body, and creating, as training data, the two-dimensional image to which information about the key point is added; A model generation process of performing machine learning using the training data to generate a machine learning model that outputs the key points from the two-dimensional image; Run the command, In the teacher data creation process, when image coordinates of two or more vertices of the rectangular shape are designated as the key points of the moving object, A model generation program in which, in the teacher data creation process, the key points in the two-dimensional image are indicated in a predetermined order.

7. an inference step of inferring the key points of the moving object from the two-dimensional image by using the machine learning model generated by the model generation method according to any one of claims 1 to 4; an output step of performing a calculation required to estimate the posture of the moving object on the key points; A moving object posture estimation method comprising:

8. an inference unit that infers the key points of the moving object from the two-dimensional image by using the machine learning model generated by the model generation method according to any one of claims 1 to 4; an output unit that performs a calculation required to estimate the posture of the moving object for the key points; A moving object posture estimation device comprising:

Images