DATA PROCESSING APPARATUS, DATA PROCESSING METHOD, AND DATA PROCESSING PROGRAM
The data processing device integrates data features from multiple surveillance cameras by aligning feature vectors based on real-space positional relationships, addressing the challenge of varying camera positions and target movement to achieve accurate posture and shape estimation.
Patent Information
- Application Number
- JP2020192448
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-11-19
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2040-11-19
AI Technical Summary
Integrating multiple estimation results from images obtained from different surveillance cameras to achieve stable and accurate estimation of a person's posture and body shape is challenging due to varying positional relationships and visibility, especially when cameras are fixed and the target moves within the surveillance space.
A data processing device and method that extracts and integrates data features from multiple viewpoints by considering the positional relationship between measurement points and the object, using a feature extraction unit, shape conversion unit, and feature integration unit to align and combine feature vectors based on real-space positional relationships.
Enables accurate integration of feature amounts from multiple viewpoints, allowing for precise estimation of the person's shape and posture despite changing positional relationships, enhancing estimation accuracy even when cameras are fixed and the target moves.
Smart Images

Figure 0007594889000001 
Figure 0007594889000002 
Figure 0007594889000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a data processing device, a data processing method, and a data processing program, and more particularly to a data processing device, a data processing method, and a data processing program for extracting feature quantities from data measured from a plurality of measurement points. [Background technology]
[0002] In the task of estimating the position and shape of an object from an image, it is possible to obtain more accurate results by using multiple images taken from different positions rather than using only one image. In particular, if the position and orientation of the camera that took each image (hereafter, these will be called extrinsic parameters) are known, it becomes possible to calculate the position of the object using the so-called principle of triangulation. Also, something that is hidden from one camera and cannot be seen can sometimes be observed from another camera, and in such cases, accuracy can be improved by using multiple images.
[0003] In recent years, research has been actively conducted into estimating the distance to an object, its shape, and its posture using machine learning. For example, Non-Patent Document 1 describes an algorithm that uses deep learning to estimate the posture and body type of an object from a single image. In addition, Non-Patent Document 2 describes that in order to estimate the type of furniture or the like from multiple images, multiple cameras are placed at equal intervals around the object to capture images, thereby enabling shape reconstruction with higher accuracy than when only one camera is used. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] Kolotouros, Nikos, Georgios Pavlakos, and Kostas Daniilidis. “Convolutional Mesh Regression for Single-Image Human Shape Reconstruction.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. [Non-Patent Document 2] Su, Hang, et al. “Multi-view convolutional neural networks for 3d shape recognition.” Proceedings of the IEEE international conference on computer vision. 2015. Summary of the Invention [Problem to be solved by the invention]
[0005] We consider applying such technology to estimate the posture and body type of a person based on images obtained from multiple surveillance cameras. In this case, for example, a method of obtaining estimation results by applying Non-Patent Document 1 to each image is considered.
[0006] This method obtains multiple estimation results with large errors, as they are estimated using only images from a single camera, making it a difficult problem to determine how to integrate the multiple estimation results.
[0007] For example, it would be possible to integrate the results while taking into account the relative positions of each surveillance camera and the target, as well as how well the target is captured in the image, but it is difficult to create an integration algorithm that can produce stable, highly accurate estimation results under a variety of conditions.
[0008] In addition, because surveillance cameras are fixed to walls or ceilings in advance, when a person moves within the monitored space, it is difficult to constantly maintain the positional relationship between the target and the camera as in Non-Patent Document 2.
[0009] Therefore, an object of the present invention is to provide a data processing device, a data processing method, and a data processing program that can integrate features of measurement data from multiple measurement points while taking into account the relationship between the measurement points and the object. [Means for solving the problem]
[0010] In order to achieve the above-mentioned object, the data processing device of the present invention is configured to include a feature extraction unit that extracts data features consisting of elements arranged in a predetermined dimension from each of data measured from a plurality of measurement points, and a feature integration unit that determines a correspondence between elements between the data features of the data from different measurement points based on a positional relationship in real space between each of the plurality of measurement points and an object represented by the data, and integrates corresponding elements to integrate the data features of the data from the different measurement points.
[0011] According to the data processing device of the present invention, the feature extraction unit extracts, for each piece of data measured from a plurality of measurement points, data features made up of elements arranged in a predetermined dimension from the data.
[0012] Then, the feature integration unit determines the correspondence between elements of the data features of the data at different measurement points based on the positional relationship in real space between each of the multiple measurement points and the object represented by the data, and integrates the data features of the data at the different measurement points by integrating corresponding elements.
[0013] In this way, by determining the correspondence between elements of data features of different measurement points based on the positional relationship in real space between each of the multiple measurement points and the object represented by the data, and integrating corresponding elements, it is possible to integrate the features of measurement data from the multiple measurement points while taking into account the relationship between the measurement points and the object.
[0014] In addition, the data feature is a circulant vector or matrix in which elements are arranged in a circulant manner, and the feature integration unit is capable of integrating the data features of the data of the different measurement points by shifting the elements of the data feature in accordance with the shift, the correspondence being a shift between corresponding elements.
[0015] The deviation of the corresponding elements may be a deviation according to a relative angle of a measurement direction from each of the different measurement points to the object.
[0016] In addition, the data processing device may further include an estimation unit that outputs an estimation result based on features obtained by integrating the data features by the feature integration unit, and a machine-learned learning model functions at least as the estimation unit, and the learning model is machine-learned using the data, information defining the positional relationship, the position of the target object, and the correct estimation result as learning data.
[0017] In a data processing method according to the present invention, a feature extraction unit extracts data features consisting of elements arranged in a predetermined dimension from each of data measured from a plurality of measurement points, and a feature integration unit determines a correspondence between elements between the data features of the data of different measurement points based on a positional relationship in real space between each of the plurality of measurement points and an object represented by the data, and integrates corresponding elements to integrate the data features of the data of the different measurement points.
[0018] The data processing program of the present invention is a program for causing a computer to function as a feature extraction unit that extracts data features consisting of elements arranged in a predetermined dimension from each of data measured from a plurality of measurement points, and a feature integration unit that determines a correspondence between elements between data features of the data from different measurement points based on the positional relationship in real space between each of the plurality of measurement points and the object represented by the data, and integrates corresponding elements to integrate the data features of the data from the different measurement points. Effect of the Invention
[0019] According to the data processing device, data processing method, and data processing program of the present invention, it is possible to obtain the effect that feature quantities of measurement data from a plurality of measurement points can be integrated taking into account the relationship between the measurement points and the object. [Brief description of the drawings]
[0020] [Figure 1] 1 is a schematic diagram showing a configuration of an image processing system according to an embodiment of the present invention; [Diagram 2] 1 is a schematic diagram illustrating a configuration of an image processing device according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram showing an example of a plurality of monitoring images. [Figure 4] FIG. 13 is a schematic diagram showing a shape feature vector; [Diagram 5] FIG. 13 is a schematic diagram showing an example of integrating shape feature vectors; [Figure 6] 1 is a schematic diagram showing a configuration of a learning device according to an embodiment of the present invention; [Figure 7] 5 is a flowchart showing the operation of a learning process by the learning device according to the embodiment of the present invention. [Figure 8] 4 is a flowchart showing an image processing operation by the image processing device according to the embodiment of the present invention. [Figure 9] FIG. 1 is a diagram showing an image of how the shape of a person is restored from multiple surveillance images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0021] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, the present invention will be described by taking as an example a case where the present invention is applied to an image processing system in which a plurality of image capturing devices are present in a monitored space and the posture, body shape, and other shape of a person moving within the space are estimated.
[0022] <System configuration> Hereinafter, the configuration of an embodiment of the present invention will be described with reference to FIG. 1 which shows a schematic configuration of an image processing system 1000 to which the present invention is applied.
[0023] (Image processing system 1000) The image processing system 1000 includes a plurality of image capturing devices 1100, a network 1200, an image processing device 100, and a learning device 150. The image processing device 100 is an example of a data processing device.
[0024] (imaging device 1100) The multiple image capturing devices 1100 capture surveillance images from multiple viewpoints. Here, the viewpoints are an example of measurement points. Specifically, the multiple image capturing devices 1100 are surveillance cameras installed at multiple installation locations for the purpose of monitoring a specific surveillance space. The surveillance images captured by the image capturing devices 1100 are transmitted to the image processing device 100.
[0025] (Network 1200) The network 1200 is a line used for transmitting and receiving data between a plurality of imaging devices 1100, image processing devices 100, and learning devices 150. A LAN (Local Area Network) or a public line such as the Internet can be used as the network 1200 of the present invention. It is desirable to take security measures for messages on the network 1200, such as encrypting the messages using known VPN technology or the like.
[0026] (Image processing device 100) The image processing device 100 is composed of a CPU, GPU, MPU, peripheral circuits, terminals, various memories, etc., and receives surveillance images transmitted from a plurality of image capturing devices 1100 and estimates the shapes of people appearing in the surveillance images.
[0027] As shown in FIG. 2, the image processing device 100 functionally comprises an input unit 10, a calculation unit 20, and an output unit 50.
[0028] The input unit 10 accepts the received multiple surveillance images. Here, the positions of the multiple image capturing devices 1100 and the position of the target person appearing in the surveillance images are assumed to be known, and multiple surveillance images are obtained as shown in Fig. 3. The position of the target person may be measured by tracking using images, or a laser sensor, an infrared sensor, an ultrasonic sensor, radio wave positioning, or the like may be used.
[0029] The calculation unit 20 is equipped with a neural network that has been trained in advance, and inputs a plurality of surveillance images to the neural network, which estimates the shapes of people appearing in the surveillance images and outputs them via the output unit 50.
[0030] Below, a detailed description will be given of each of the feature extraction unit 22, the shape conversion unit 24, the feature integration unit 26, and the estimation unit 28 that configure the calculation unit 20. The feature extraction unit 22 and the shape conversion unit 24 are examples of a feature extraction unit.
[0031] The feature extraction unit 22 extracts a feature vector for each of the surveillance images taken from multiple viewpoints. Specifically, each surveillance image is converted into a feature vector by a convolutional neural network. At this time, a mask image of a person in the surveillance image may be created by using an instance segmentation technique or the like, and the mask image may be input to the convolutional neural network at the same time. In this case, it is possible to reduce the possibility that the estimation accuracy will decrease due to the diversity of the background, and it becomes possible to select a person to be processed when multiple people are captured in the surveillance image. In addition, a person area of a target person may be cut out from the surveillance image, and a feature vector may be extracted for each of the cut-out person areas. In addition, a feature at a position corresponding to the person area in the feature vector extracted from the entire surveillance image may be extracted, and used as a feature vector to be used in subsequent processing.
[0032] The shape conversion unit 24 converts the feature vector extracted for each surveillance image into a shape feature vector, which is a circulating vector in which elements are arranged in a circulating manner. Specifically, a circulating vector is a vector in which elements are arranged along a shape having an axis that overlaps with the shape before rotation even when rotated by any angle. The shape feature vector is an example of a data feature.
[0033] For example, the shape conversion unit 24 creates a shape-based feature vector 90 in which elements expressing features are distributed in a ring shape based on the feature vector (FIG. 4). The elements are arranged so that the measurement target (subject) is located at the center along the ring shape as shown in FIG. 4, and shape conversion is performed so that the elements expressing the features of the measurement target are arranged by azimuth angle.
[0034] For this operation, for example, a fully connected layer in Deep Learning may be used, or the shape of the measurement target may be approximated by a cylinder, and the feature vector may be arranged on the cylinder using the correspondence when the cylinder is projected onto the image, and the average may be obtained in the height direction of the cylinder. Here, it should be noted that the feature vector of the feature extracting unit 22 is calculated without using the external parameters of the camera device 1100, and the feature vector does not include information on the position of the camera device 1100. Therefore, when creating the shape feature vector 90, for example, if the shape feature vector 90a in FIG. 5 corresponds to the image captured by the camera device 1100a and the shape feature vector 90b corresponds to the image captured by the camera device 1100b, the shape feature vectors 90a,b are created by performing shape conversion so that elements (elements connected by solid lines extending from the camera devices 1100a,b) representing the features at the positions where the camera device 1100 and the person to be measured are closest to each other are arranged. In this way, a method that does not use the external parameters of the camera device 1100 for shape conversion is used.
[0035] The feature amount integration unit 26 determines the shift of corresponding elements as the correspondence between elements of the shape feature amount vectors 90 of the surveillance images from different viewpoints based on the positional relationship in real space between each of the multiple viewpoints and the person depicted in the surveillance image, and operates to shift the elements of the shape feature amount vector 90 according to the shift. Here, the shift of corresponding elements is a shift according to the relative angle of the shooting direction from each of the different viewpoints toward the position of the person.
[0036] Specifically, the feature amount integration unit 26 rearranges the elements of the shape feature amount vector 90 based on the external parameters of the image capturing device 1100 and the position of the person.
[0037] For example, first, the ring of the shape feature vector 90 is placed on a reference surface such as the floor surface, and the ring of the shape feature vector 90 is rotated so that the element of the shape feature vector 90 that represents the azimuth angle of the camera 1100, i.e., the ring element that represents the feature amount at the position where the camera 1100 and the person to be measured are closest to each other, actually faces the azimuth angle of the camera 1100. The amount of rotation can be calculated from the position of the person in the surveillance image and the external parameters of the camera 1100.
[0038] The rotation operation is equivalent to lining up feature vectors, then shifting elements to the right or left, and moving elements that protrude from one end to the other. If the shift amount is not an integer, continuous rotation can be expressed by using a weighted average of the previous and next elements.
[0039] In this way, in the shape feature vector 90, the azimuth angle relationship from the image capture device 1100 to the person depicted in the surveillance image can be expressed as the rotation of a ring (FIG. 5). Here, the azimuth angle relationship is an example of a positional relationship in real space. In this case, since the depression angle information is lost, the depression angle may be added to the dimension of the shape feature vector 90, or the depression angle may be expressed by two functions, a cosine function and a sine function, and the two values may be added to the dimension of the shape feature vector 90. In addition, the distance between the image capture device 1100 and the person depicted in the surveillance image can be reflected by a method such as adding an element representing the distance value or its reciprocal to the shape feature vector 90, or making the viewing angle of the input image constant so that distant objects appear smaller.
[0040] In addition, the feature integration unit 26 integrates corresponding elements of each shape feature vector 90 of the surveillance image, thereby integrating the shape feature vectors 90 of the surveillance images from different viewpoints, and outputs a single shape feature vector 90.
[0041] For this operation, for example, a method of taking the sum of elements of each shape feature vector 90 after the rearrangement operation to create one shape feature vector 90 including information of all the shape feature vectors 90 can be used. Alternatively, an operation may be performed in which two shape feature vectors 90 after the rearrangement operation are overlapped and one-dimensional convolution is performed in the circumferential direction to output one shape feature vector 90. Since these two methods are methods for creating one shape feature vector 90 from two shape feature vectors 90, by applying these methods multiple times, one shape feature vector 90 can be output regardless of the number of shape feature vectors input. When the number of image capture devices 1100 does not change, shape feature vectors 90 corresponding to the number of image capture devices 1100 may be integrated simultaneously.
[0042] The estimation unit 28 receives as input one shape feature vector 90 output by the feature integration unit 26, estimates the shape of a person depicted in a surveillance image, and outputs the result via the output unit 50. Specifically, one shape feature vector 90 is input to a neural network, and an output representing the shape of a person is obtained.
[0043] For example, parameters expressing the 3D shape and posture of a person are estimated using a fully connected layer. To link the parameters with the posture and shape of the person, for example, the method described in Non-Patent Document 3 may be used, or all vertices of a mesh model expressing the shape of the person may be estimated and obtained.
[0044] [Non-Patent Document 3] Loper, Matthew, et al. "SMPL: A skinned multi-person linear model." ACM transactions on graphics (TOG) 34.6 (2015): 1-16.
[0045] (Learning device 150) The parameters of the model used in the image processing device 100 are learned by the learning device 150 .
[0046] The learning device 150 is composed of a CPU, GPU, MPU, peripheral circuits, terminals, various memories, etc., and accepts multiple surveillance images taken from multiple viewpoints, the shapes of people depicted in the multiple surveillance images, external parameters representing the position and attitude of the imaging device 1100 when the images were taken, and position information of the people, and learns model parameters for estimating the shapes of people appearing in the surveillance images.
[0047] As shown in FIG. 6, the learning device 150 functionally comprises an input unit 60 and a calculation unit .
[0048] The input unit 60 receives multiple surveillance images captured from multiple viewpoints, the shape of a person depicted in the multiple surveillance images, external parameters representing the position and attitude of the imaging device 1100 at the time the images were captured, and position information of the person.
[0049] For example, as learning data, a plurality of images of a person photographed from a plurality of viewpoints, a plurality of sets of external parameters representing the position and posture of the photographing device 1100 when the photographs were taken, and shape and position information of the person are prepared, and the input unit 60 receives the plurality of learning data. These can be prepared, for example, by using CG.
[0050] Hereinafter, each part of the learning unit 72 and the model storage unit 74 constituting the calculation unit 70 will be described in detail.
[0051] The learning unit 72 learns model parameters for estimating the shape of a person appearing in a surveillance image based on a plurality of learning data.
[0052] Specifically, since the operations used in the feature extraction unit 22, shape conversion unit 24, feature integration unit 26, and estimation unit 28 of the image processing device 100 are all differentiable, a commonly used deep learning learning algorithm is used to learn the parameters of a neural network model that functions as the entirety of the feature extraction unit 22, shape conversion unit 24, feature integration unit 26, and estimation unit 28.
[0053] For example, based on multiple surveillance images, external parameters, and person position information of the learning data, various parameters can be repeatedly updated so as to minimize the error function between the estimation results output by the neural network model functioning as a whole of the feature extraction unit 22, shape conversion unit 24, feature integration unit 26, and estimation unit 28 and the shape data created by CG as the correct answer data, thereby obtaining the desired parameters of the neural network model.
[0054] The model storage unit 74 stores the parameters of the model learned by the learning unit 72 .
[0055] <Image processing system operation> Hereinafter, the operation of the image processing system 1000 to which the present invention is applied will be described with reference to the flowcharts shown in Figures 7 and 8. Note that the description will be given taking as an example a case in which a plurality of pieces of learning data prepared in advance are input to the learning device 150 by the input unit 60.
[0056] 7 is executed in advance. In the learning process, a neural network model that functions as a whole of the feature extraction unit 22, the shape conversion unit 24, the feature integration unit 26, and the estimation unit 28 of the image processing device 100 is used to estimate the shape of a person for each piece of learning data (step S100).
[0057] Then, the parameters of the neural network model are updated so as to minimize the error function between the estimation result in step S100 and the correct data (step S102).
[0058] Then, it is determined whether or not a predetermined iteration end condition is satisfied (step S104). If the predetermined iteration end condition is not satisfied, the process returns to step S100, whereas if the predetermined iteration end condition is satisfied, the process proceeds to step S106.
[0059] The iteration end condition may be that the value of the error function has converged or that the number of iterations has reached an upper limit.
[0060] Then, the parameters of the finally trained neural network model are stored in the model storage unit 74 (step S106), and the training process ends.
[0061] Then, the parameters of the neural network model that functions as the feature extraction unit 22, the shape conversion unit 24, the feature integration unit 26, and the estimation unit 28 as a whole, learned by the learning device 150, are set in the image processing device 100.
[0062] The image processing device 100, which has received a plurality of surveillance images from a plurality of image capturing devices 1100, performs the image processing shown in Fig. 8. The operation of the image processing device 100 in Fig. 8 described below is executed each time a set of a plurality of surveillance images captured by the plurality of image capturing devices 1100 is received. When cutting out a person area of a target person from a surveillance image and extracting a feature vector for each of the cut-out person areas, the person area of the target person is cut out from each surveillance image each time a set of a plurality of surveillance images captured by the plurality of image capturing devices 1100 is received, and the image processing shown in Fig. 8 is performed on the image of the cut-out person area.
[0063] First, the feature amount extracting unit 22 extracts a feature amount vector for each of a plurality of monitoring images captured from a plurality of viewpoints (step S110).
[0064] The shape conversion unit 24 converts the feature vector extracted for each of the multiple surveillance images into a shape feature vector 90 in which elements are arranged in a cyclic manner (step S112).
[0065] Based on the positional relationship in real space between each of the multiple viewpoints and the person depicted in the surveillance image, the feature amount integration unit 26 determines the shift of corresponding elements as the correspondence between elements of the shape feature amount vectors 90 of the surveillance images from different viewpoints, and operates to shift the elements of the shape feature amount vectors 90 according to the shift (step S114). At this time, if the shift amount is not an integer, an interpolation operation such as linear interpolation is performed based on adjacent elements to calculate the value of the element after rotation.
[0066] In addition, the feature integration unit 26 integrates the shape feature vectors 90 of the surveillance images from different viewpoints by integrating corresponding elements of each shape feature vector 90 of the surveillance images, and outputs a single shape feature vector 90 (step S116).
[0067] The estimation unit 28 receives as input one shape feature vector 90 output by the feature integration unit 26, estimates the shape of a person depicted in a surveillance image, and outputs the result via the output unit 50 (step S118). Specifically, one shape feature vector 90 is input to a fully connected layer, and an output representing the shape of a person is obtained.
[0068] As described above, in the image processing device 100 according to the embodiment of the present invention, the corresponding relationship between elements of the shape feature vectors 90 of the surveillance images from different viewpoints is determined based on the positional relationship in real space between each of the multiple viewpoints and the person depicted in the surveillance image, and the corresponding elements are integrated, so that the feature amounts of the surveillance images from multiple viewpoints can be integrated while considering the relationship between the viewpoints and the person. For example, as shown in FIG. 5, the feature amounts of the surveillance images obtained from multiple image capture devices can be integrated while considering the positional relationship between the image capture devices and the person. Therefore, even in a situation where the positional relationship between the image capture devices and the person is constantly changing, for example, a situation where a person is walking around in a surveillance space, the feature amount vectors of the surveillance images from other viewpoints can be restored by effectively integrating the feature amount vectors from the surveillance images of the multiple image capture devices that captured the person at the same time, and the shape of the person can be accurately estimated (see FIG. 9(a)). FIG. 9(a) shows an image in which the shape of a person is estimated from the surveillance images of multiple image capture devices, and the surveillance images from other viewpoints are restored using the estimated shape.
[0069] <Modification> Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments. For example, in this embodiment, the conversion to a shape feature vector 90 in which elements are arranged in a ring shape has been described as an example, but the present invention is not limited to this. For example, the conversion to a shape feature vector 90 in which elements are arranged on a cylinder or sphere may be performed. In this case, the elements can be integrated by taking the sum of the elements in the same way. Furthermore, if convolution is used for integration, integration is also possible by using two-dimensional convolution.
[0070] Furthermore, if a sphere, for example, is used as the shape of the shape feature vector 90, unlike a ring, the sphere can be rotated three-dimensionally, so not only the azimuth angle from the person to the camera, but also the depression angle can be expressed. In this way, it is preferable for a circulating vector to be arranged along a shape with an axis that overlaps with the shape before rotation, even when rotated at any angle.
[0071] The imaging device may be moving. In this case, it is necessary to sequentially acquire external parameters at the time of imaging by combining an IMU (Inertial Measurement Unit) with the imaging device.
[0072] Although the above description is directed to an example in which the estimation unit estimates the shape of a person, the present invention is not limited to this example. For example, the estimation unit may estimate the shape of a general object, estimate three-dimensional key points of a person, or perform person identification.
[0073] Further, the description has been given with respect to an example in which each unit of the image processing device 100 uses a neural network, but the present invention is not limited to this. For example, the feature extraction unit 22 may extract HOG (Histogram of Oriented Gradients) features from an image, the shape conversion unit 24 may convert the feature vector 90 into a shape with elements representing the HOG features attached to a cylindrical surface, the feature integration unit 26 may integrate the feature vector 90 with shape, and the estimation unit 28 may estimate the shape by regression using a support vector machine (SVM). In this way, the present invention may be applied to other machine learning methods other than deep learning. Furthermore, each of the shape conversion unit 24 and the feature integration unit 26 can be realized without using machine learning.
[0074] In addition, the case where the feature amounts of the monitoring images taken from a plurality of viewpoints are integrated has been described as an example, but the present invention is not limited to this. For example, the feature amounts of the 3D point clouds of the person area of the distance images taken from a plurality of viewpoints may be integrated (FIG. 9(b)). FIG. 9(b) shows an image of restoring a 3D point cloud of the person area from another viewpoint from the 3D point clouds of the person area of the distance images taken by a plurality of distance image taking devices. In addition, the feature amounts of the measurement data measured from a plurality of measurement points using a measurement device other than the distance image taking device may be integrated. For example, a microphone may be used to measure sound from a plurality of measurement points, or a LiDAR may be used to measure a point cloud from a plurality of measurement points. In addition, a posture may be estimated from the measurement data measured from a plurality of measurement points using a wifi radio wave.
[0075] In addition, in the present embodiment, a case where a person is measured as a measurement object has been described as an example, but the present invention is not limited to this. An object other than a person may be measured as a measurement object.
[0076] Furthermore, the learning unit 72 may be provided in the image processing device 100 instead of being provided in a learning device 150 separate from the image processing device 100 .
[0077] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]
[0078] 10, 60 Input section 20, 70 calculation section 22 Feature Extraction Unit 24 Shape conversion section 26 Feature Integration Unit 28 Estimation part 50 Output section 72 Learning Department 74 Model Memory Section 100 Image processing device 150 Learning Device 1000 Image Processing System 1100 Imaging device
Claims
1. a feature extraction unit that extracts data feature values consisting of elements arranged in a predetermined dimension from each of data measured from a plurality of measurement points; a feature integration unit that determines a correspondence relationship between elements of data feature amounts of the data at different measurement points based on a positional relationship in real space between each of the plurality of measurement points and an object represented by the data, and integrates corresponding elements to integrate the data feature amounts of the data at different measurement points; 2. A data processing device comprising:
2. the data feature is a circulant vector or matrix in which elements are arranged circulantly, The data processing device according to claim 1 , wherein the feature integration unit integrates the data feature quantities of the data at the different measurement points by shifting the elements of the data feature quantities according to the shift, as the correspondence relationship being a shift between corresponding elements.
3. 3. The data processing apparatus according to claim 2, wherein the deviation of the corresponding elements is a deviation corresponding to a relative angle of the measurement direction from each of the different measurement points to the object.
4. further including an estimation unit that estimates a shape of the object based on a feature amount obtained by integrating the data feature amounts by the feature amount integration unit; a machine-learned learning model functions as the feature extraction unit, the feature integration unit, and the estimation unit; A data processing device according to any one of claims 1 to 3, wherein the learning model is machine-learned using the data, information specifying the positional relationship, the position of the object, and shape data that is the correct estimation result as learning data.
5. the feature extraction unit extracts, for each of the data measured from the plurality of measurement points, a data feature consisting of elements arranged in a predetermined dimension from the data; The feature integration unit determines a correspondence relationship between elements of the data feature amounts of the data at different measurement points based on a positional relationship in real space between each of the plurality of measurement points and the object represented by the data, and integrates the data feature amounts of the data at the different measurement points by integrating corresponding elements. Data processing methods.
6. Computer, a feature extraction unit that extracts, for each of the data measured from a plurality of measurement points, a data feature consisting of elements arranged in a predetermined dimension from the data; and a feature integration unit that determines a correspondence relationship between elements of data feature amounts of the data at different measurement points based on a positional relationship in real space between each of the plurality of measurement points and an object represented by the data, and integrates the data feature amounts of the data at the different measurement points by integrating corresponding elements; A data processing program that functions as a
Citation Information
Patent Citations
Three-dimensional model retrieval method based on LSTM network multi-modal information fusion
CN110163091A
Image recognition device, image recognition method, and program
JP2013218605A
Shape estimating apparatus
JP2018136632A
Imaging device, information processing device, and imaging system
JP2019049572A
Parameter optimization device, parameter optimization method and program
JP2019079487A
