Information processing device and program

Abstract

To improve the accuracy of object recognition. [Solution] The information processing device 1 includes a point cloud data acquisition unit 132 that acquires point cloud data containing multiple three-dimensional coordinates corresponding to each of multiple objects; a division unit 133 that divides the point cloud data into partial point cloud data, which are point cloud data representing each of the multiple objects, based on the acquired point cloud data; a position relationship identification unit 134 that identifies the position relationship between the object represented by each of the divided partial point cloud data and an object different from that object; and a recognition unit 135 that recognizes the object corresponding to each of the divided partial point cloud data based on each of the divided partial point cloud data and the position relationship identified for that partial point cloud data.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, object recognition has been performed based on point cloud data including a plurality of three-dimensional coordinates obtained by measurement (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a part of the data of the point cloud data corresponding to an object is insufficient due to a plurality of objects overlapping, there has been a problem that the object cannot be accurately recognized.

[0005] Therefore, the present invention has been made in view of these points, and an object thereof is to improve the recognition accuracy of an object.

Means for Solving the Problems

[0006] An information processing device according to a first aspect of the present invention includes: an acquisition unit that acquires point cloud data including a plurality of three-dimensional coordinates corresponding to each of a plurality of objects; a division unit that divides the point cloud data acquired by the acquisition unit into partial point cloud data which are point cloud data representing each of the plurality of objects; a position relationship identification unit that identifies the position relationship between each of the plurality of partial point cloud data divided by the division unit and an object different from that object; and a recognition unit that recognizes the object corresponding to each of the plurality of partial point cloud data based on each of the plurality of partial point cloud data divided by the division unit and the position relationship identified by the position relationship identification unit with respect to the partial point cloud data.

[0007] The position relationship identification unit may identify the position relationship corresponding to the object corresponding to the partial point cloud data by inputting the partial point cloud data divided by the division unit into a position relationship identification model that outputs position relationship information indicating the position relationship when the partial point cloud data is input, and by acquiring the position relationship information output from the position relationship identification model.

[0008] The information processing device further includes a learning unit that generates a positional relationship identification model by learning partial point cloud data corresponding to an object whose positional relationship has been identified and positional relationship information indicating the positional relationship as training data. The positional relationship identification unit may identify the positional relationship corresponding to the object indicated by the partial point cloud data by inputting the partial point cloud data divided by the division unit into the positional relationship identification model generated by the learning unit and obtaining the positional relationship information output from the positional relationship identification model.

[0009] The recognition unit may, upon receiving the partial point cloud data and positional relationship information indicating the positional relationship, input the partial point cloud data divided by the division unit and the positional relationship information indicating the positional relationship identified by the positional relationship identification unit to an object recognition model that outputs a label for identifying the recognized object, and acquire the label output from the object recognition model to recognize the object corresponding to the partial point cloud data.

[0010] The information processing device further includes a learning unit that generates the object recognition model by learning the partial point cloud data corresponding to a previously recognized object, the positional relationship corresponding to the object, and the label corresponding to the object as training data. The recognition unit may recognize the object corresponding to the partial point cloud data by inputting the partial point cloud data divided by the division unit and positional relationship information indicating the positional relationship identified by the positional relationship identification unit for the partial point cloud data into the object recognition model generated by the learning unit, and by obtaining the label output from the object recognition model.

[0011] The information processing device further includes a learning unit that generates a partitioning model that outputs multiple partial point cloud data in response to input point cloud data by learning point cloud data corresponding to a plurality of objects that have been recognized in advance and a plurality of partial point cloud data corresponding to each of the plurality of objects as training data, and the partitioning unit may divide the point cloud data into a plurality of partial point cloud data by inputting the point cloud data acquired by the acquisition unit to the partitioning model generated by the learning unit and acquiring one or more of the partial point cloud data from the partitioning model.

[0012] The positional relationship identification unit may specify the positional relationship as any of the following: a positional relationship in which the object does not overlap with the other object; a positional relationship in which at least a part of the object is hidden by the other object being adjacent to the object; a positional relationship in which the other object overlaps the top of the object; a positional relationship in which a part of the object is covered by the other object; or a positional relationship in which the object contains the other object. The positional relationship identification unit may identify the density of the point cloud corresponding to the partial point cloud data as the positional relationship.

[0013] The information processing device may further include an attribute identification unit that identifies at least one of the following as attributes of the object based on the partial point cloud data corresponding to the object recognized by the recognition unit: the type of the object, the position of the object, the direction the object is facing, the shape of the object, the area of ​​the object, the volume of the object, and the state of the object; and a storage control unit that causes the attribute of the object identified by the attribute identification unit to be stored in a storage unit.

[0014] The acquisition unit acquires the point cloud data for each of the multiple different time periods; the division unit divides the point cloud data for each of the multiple different time periods acquired by the acquisition unit into multiple partial point cloud data; the position relationship identification unit identifies the position relationship for each of the multiple partial point cloud data divided for each of the multiple different time periods; the recognition unit recognizes an object corresponding to each of the multiple partial point cloud data corresponding to each of the multiple different time periods based on each of the multiple partial point cloud data corresponding to each of the multiple different time periods and the position relationship identified by the position relationship identification unit for the said partial point cloud data corresponding to each of the multiple different time periods; the attribute identification unit identifies the attributes of the object recognized by the recognition unit in relation to the multiple different time periods; and the information processing device may further include a detection unit that detects changes in the attributes of the object based on the attributes of the object identified by the attribute identification unit in relation to the multiple different time periods, and a notification unit that notifies of the changes in the attributes of the object detected by the detection unit.

[0015] A second aspect of the present invention relates to an information processing method which includes the steps of: acquiring point cloud data, which includes a plurality of three-dimensional coordinates corresponding to each of a plurality of objects, executed by a computer; dividing the acquired point cloud data into partial point cloud data, which are point cloud data representing each of the plurality of objects, based on the acquired point cloud data; identifying the positional relationship between the object represented by each of the divided partial point cloud data and an object different from that object; and recognizing the object corresponding to each of the plurality of partial point cloud data based on each of the divided partial point cloud data and the positional relationship identified for each of the partial point cloud data.

[0016] A program according to a third aspect of the present invention causes a computer to function as: an acquisition unit that acquires point cloud data including a plurality of three-dimensional coordinates corresponding to each of a plurality of objects; a division unit that divides the point cloud data acquired by the acquisition unit into partial point cloud data which are point cloud data representing each of the plurality of objects; a position relationship identification unit that identifies the position relationship between each of the plurality of partial point cloud data divided by the division unit and an object different from that object; and a recognition unit that recognizes the object corresponding to each of the plurality of partial point cloud data based on each of the plurality of partial point cloud data divided by the division unit and the position relationship identified by the position relationship identification unit with respect to the partial point cloud data. [Effects of the Invention]

[0017] According to the present invention, the accuracy of object recognition can be improved. [Brief explanation of the drawing]

[0018] [Figure 1] This is a diagram illustrating the overview of an information processing device. [Figure 2] This diagram shows the functional configuration of an information processing device. [Figure 3] This diagram illustrates six different spatial relationships. [Figure 4] This figure shows an example of object information. [Figure 5] This is a flowchart showing the process flow for an information processing apparatus to detect changes in the state of an object.

Embodiments for Carrying Out the Invention

[0019] [Overview of Information Processing Apparatus 1] FIG. 1 is a diagram showing an overview of information processing apparatus 1. The information processing apparatus 1 is a computer that recognizes an object from point cloud data including a plurality of three-dimensional coordinates corresponding to each of a plurality of objects. The information processing apparatus 1 is communicably connected via a communication network such as a wireless LAN or the Internet to a photographing apparatus 2 that photographs the inside of a facility including the object to be recognized, and a terminal 3 used by an administrator who manages the facility.

[0020] The information processing apparatus 1, for example, acquires video data obtained by photographing from a photographing apparatus 2 mounted on a moving body such as a vehicle or a drone that moves inside a facility including the object to be recognized, at a plurality of viewpoint positions inside the facility ((1) in FIG. 1). The information processing apparatus 1 acquires point cloud data including a plurality of three-dimensional coordinates corresponding to each of a plurality of objects, corresponding to each of a plurality of different viewpoint positions, based on a plurality of images included in the acquired video data ((2) in FIG. 1).

[0021] The information processing apparatus 1 divides the point cloud data acquired from the video data into a plurality of partial point cloud data which are a group of point cloud data indicating each of the plurality of objects ((3) in FIG. 1). Details of the method for dividing the point cloud data group into a plurality of partial point cloud data will be described later. Among the plurality of divided partial point cloud data, there may be included partial point cloud data in which another object is arranged between the object and the photographing position of the photographing apparatus 2 and a part of the object is not represented because the entire object cannot be photographed.

[0022] The information processing device 1 identifies the positional relationship between the object represented by each of the multiple sub-point cloud data points separated from the point cloud data and other objects (Figure 1, (4)). The information processing device 1 identifies the positional relationships of multiple objects by using a positional relationship identification model, which was created by learning the relationship between sub-point cloud data and positional relationships in advance, and which outputs positional relationship information indicating the positional relationship when sub-point cloud data is input. The information processing device 1 inputs the separated sub-point cloud data into the generated positional relationship identification model and obtains the positional relationship information output by the positional relationship identification model, thereby identifying the positional relationship corresponding to the object corresponding to the sub-point cloud data.

[0023] The information processing device 1 recognizes an object corresponding to a partial point cloud data based on the partial point cloud data divided from the point cloud data and the positional relationship identified with respect to the partial point cloud data (Figure 1, (5)). The information processing device 1 recognizes an object by using an object recognition model created by learning the relationship between the partial point cloud data and the object it represents, corresponding to each of multiple positional relationships between the object represented by the partial point cloud data and other objects. The information processing device 1 recognizes an object corresponding to a partial point cloud data by inputting the divided partial point cloud data and positional relationship information indicating the positional relationship identified with respect to the partial point cloud data into the object recognition model and obtaining the label output by the object recognition model.

[0024] The information processing device 1 identifies the attributes of the recognized object, such as the type of object and its location, based on the partial point cloud data of the recognized object, and stores the identified attributes (Figure 1, (6)). The information processing device 1 notifies terminal 3 of the object information, including the stored attributes (Figure 1, (7)). This allows the administrator managing the facility using terminal 3 to check the attributes of the recognized object and understand any abnormalities in the object.

[0025] Conventional object recognition processing has a problem in that it cannot correctly recognize objects when attempting to recognize them based on partial point cloud data in which only a part of the object is represented. In contrast, the information processing device 1 identifies the positional relationship between the object represented by the partial point cloud data and other objects, and recognizes the object corresponding to the partial point cloud data based on the identified positional relationship between the partial point cloud data and other objects. By doing so, the information processing device 1 can improve the accuracy of object recognition compared to when it recognizes objects without considering the positional relationship between the object corresponding to the partial point cloud data and other objects.

[0026] [Functional configuration of the information processing device 1] Next, the details of the configuration of the information processing device 1 will be explained. Figure 2 is a diagram showing the functional configuration of the information processing device 1. The information processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The communication unit 11 is a communication interface for sending and receiving data with the imaging device 2 and the terminal 3 via a network such as the Internet.

[0027] The memory unit 12 is a storage medium for storing various types of data, and includes ROM (Read Only Memory), RAM (Random Access Memory), and hard disks. The memory unit 12 stores programs to be executed by the control unit 13. The memory unit 12 stores programs that cause the control unit 13 to function as a learning unit 131, a point cloud data acquisition unit 132, a division unit 133, a position relationship identification unit 134, a recognition unit 135, an attribute identification unit 136, a memory control unit 137, a detection unit 138, and a notification unit 139.

[0028] Furthermore, the memory unit 12 stores the division model 121, the positional relationship identification model 122, the object recognition model 123, and the object DB 124. The splitting model 121 is a neural network model program that, for example, outputs multiple subpoint cloud data corresponding to point cloud data when point cloud data is input, and is generated by the learning unit 131 through pre-training. For example, when point cloud data is input, the splitting model 121 identifies the number of objects contained in the point cloud data and calculates the probability that each feature point contained in the point cloud data corresponds to one of the identified multiple objects. For each of the multiple objects, the splitting model 121 uses the set of feature points whose calculated probability exceeds a predetermined threshold as subpoint cloud data and outputs the subpoint cloud data corresponding to each of the multiple objects. The splitting model 121 is used by the splitting unit 133 to split the point cloud data into multiple subpoint cloud data.

[0029] The positional relationship identification model 122 is a neural network model program that, for example, when partial point cloud data is input, outputs positional relationship information indicating the positional relationship between the object indicated by the partial point cloud data and other objects. It is generated by the learning unit 131 through pre-training. The positional relationship identification model 122 is used by the positional relationship identification unit 134 to identify the positional relationship between the object indicated by the partial point cloud data and other objects.

[0030] Here, we will explain the positional relationship between an object and other objects. Partial point cloud data representing an object is generated based on the video data captured by the imaging device 2. If a different object is positioned between the imaging device 2 and the object, or if another object is placed adjacent to the object, the entire object may not be captured in the video data captured by the imaging device 2, and the partial point cloud data may not represent the entire object. In response to this, the information processing device 1 identifies the positional relationship between the object and other objects.

[0031] The positional relationship between an object and another object can be classified into six types: Disjoin, Inside, Meet, Covered By, Overlap, and Contains. Figure 3 illustrates these six types of positional relationships. As shown in Figure 3, Disjoin is a positional relationship where an object does not overlap with another object, and partial point cloud data corresponding to the entire object can be obtained. Inside and Meet are positional relationships where at least a part of an object is obscured by another object's proximity, and partial point cloud data corresponding to the entire object cannot be obtained. Inside is a positional relationship where one face of an object remains visible due to obscuration by another object. Meet is a positional relationship where multiple faces of an object remain visible due to obscuration by another object.

[0032] CoveredBy indicates a position where another object is overlapping an object, and partial point cloud data corresponding to the entire object cannot be obtained. Overlap indicates a position where a part of an object is covered by another object, and partial point cloud data corresponding to the entire object cannot be obtained. Contain indicates a position where only a part of an object is visible due to another object being contained within it, and partial point cloud data corresponding to the entire object cannot be obtained.

[0033] The object recognition model 123 is a neural network model program that, for example, when partial point cloud data and positional relationship information indicating a specified positional relationship are input, outputs an object name, which is a label for identifying the object indicated by the partial point cloud data. It is generated by the learning unit 131 through pre-training. The object recognition model 123 is used by the recognition unit 135 to recognize the object indicated by the partial point cloud data.

[0034] Object DB124 stores object information, including the attributes of objects recognized from partial point cloud data. Figure 4 shows an example of object information. As shown in Figure 4, object information is information that associates at least the object ID, object name, and object attribute information.

[0035] Attribute information includes, for example, information indicating the object's location, shape, length, volume, and state. The object's location is, for example, information indicating the location of the partial point cloud data corresponding to the recognized object. The object's shape is, for example, information indicating the shape of a rectangular prism or cylinder, but is not limited to these; partial point cloud data may also be stored as the object's shape. The object's state is information indicating whether the object is normal or abnormal.

[0036] The control unit 13 is, for example, a CPU (Central Processing Unit). By executing a program stored in the memory unit 12, the control unit 13 functions as a learning unit 131, a point cloud data acquisition unit 132, a division unit 133, a position relationship identification unit 134, a recognition unit 135, an attribute identification unit 136, a memory control unit 137, a detection unit 138, and a notification unit 139.

[0037] [Model generation] First, the process of generating models used for object recognition will be explained. Before the information processing device 1 performs object recognition, the learning unit 131 performs training based on training data to generate a segmentation model 121, a positional relationship identification model 122, and an object recognition model 123, and stores these generated models in the storage unit 12.

[0038] Specifically, the learning unit 131 learns using a set of point cloud data corresponding to a plurality of pre-recognized objects and a set of multiple subpoint cloud data corresponding to each of those objects as training data, thereby generating a segmented model 121 that outputs multiple subpoint cloud data in response to point cloud data input.

[0039] For example, the point cloud data included in the training data is the point cloud data acquired by the point cloud data acquisition unit 132 described later. Also, the multiple partial point cloud data included in the training data are multiple partial point cloud data manually divided from the point cloud data included in the training data. Although the multiple partial point cloud data included in the training data are assumed to be multiple partial point cloud data manually divided, they are not limited to this, and may also be multiple partial point cloud data divided from the point cloud data by the division unit 133 described later, which are determined to be correctly divided. When the learning unit 131 generates a division model 121, it stores the generated division model 121 in the storage unit 12.

[0040] Furthermore, the learning unit 131 learns using a set of partial point cloud data corresponding to an object whose positional relationship with other objects has been identified, and positional relationship information indicating that positional relationship, as training data. As a result, when partial point cloud data is input, it generates a positional relationship identification model 122 that outputs positional relationship information indicating that positional relationship.

[0041] For example, the learning unit 131 generates a positional relationship identification model 122 by training it based on multiple training data sets, each set consisting of partial point cloud data corresponding to an object whose positional relationship with other objects has been identified, and the positional relationship identified in the partial point cloud data from among Disjoin, Inside, Meet, CoveredBy, Overlap, and Contains. Once the learning unit 131 has generated the positional relationship identification model 122, it stores the generated positional relationship identification model 122 in the storage unit 12.

[0042] Alternatively, the learning unit 131 may generate an object recognition model 123 that outputs an object name, which is a label for identifying a recognized object, when partial point cloud data and positional relationship information are input, by learning a set of partial point cloud data corresponding to a previously recognized object, the positional relationship corresponding to the object, and the label corresponding to the object as training data.

[0043] For example, the learning unit 131 generates an object recognition model 123 by training the model based on multiple training data sets, each set consisting of partial point cloud data corresponding to a pre-recognized object, partial point cloud data in which the positional relationship with other objects is specified, positional relationship information indicating that positional relationship, and an object name manually set for that object. Once the learning unit 131 has generated the object recognition model 123, it stores the generated object recognition model 123 in the storage unit 12.

[0044] [Object Recognition] Next, the process by which the control unit 13 recognizes an object will be explained. The point cloud data acquisition unit 132, division unit 133, position relationship identification unit 134, recognition unit 135, attribute identification unit 136, and memory control unit 137 of the control unit 13 work together to recognize an object and store information indicating the attributes of the recognized object in the memory unit 12. The details of the object recognition functions of the point cloud data acquisition unit 132, division unit 133, position relationship identification unit 134, recognition unit 135, attribute identification unit 136, and memory control unit 137 will be explained below.

[0045] The point cloud data acquisition unit 132 acquires point cloud data that includes multiple three-dimensional coordinates corresponding to each of multiple objects. For example, the point cloud data acquisition unit 132 acquires point cloud data from acquired video data using SfM (Structure-from-motion), a technique that generates point cloud data from multiple image data contained in captured video.

[0046] Specifically, the point cloud data acquisition unit 132 first acquires video data generated from the imaging device 2 by capturing images from various viewpoint positions, and extracts feature points contained in each of the multiple image data corresponding to the multiple viewpoint positions included in the acquired video data. The point cloud data acquisition unit 132 then performs matching of the feature points extracted from the multiple image data. From the multiple image data, the point cloud data acquisition unit 132 selects, for example, two image data sets that have a relatively large number of matching feature points, and identifies the three-dimensional coordinate positions of the feature points in a three-dimensional spatial coordinate system based on the positional relationships of the multiple matching feature points contained in the two selected image data sets.

[0047] When the point cloud data acquisition unit 132 selects image data whose imaging position and imaging direction are unknown, it estimates the imaging position and imaging direction of the selected image data by solving a PnP problem between the feature points included in the selected image data and feature points whose three-dimensional coordinate positions have already been identified. The point cloud data acquisition unit 132 calculates the three-dimensional coordinate positions of feature points included in the selected image data whose three-dimensional coordinate positions have not yet been identified by performing triangulation processing based on the imaging position and imaging direction of the selected image data and the three-dimensional coordinate positions of feature points whose three-dimensional coordinate positions have already been identified.

[0048] Subsequently, the point cloud data acquisition unit 132 adds new image data one by one and repeatedly calculates the three-dimensional coordinate positions of feature points whose three-dimensional coordinate positions have not been determined, thereby acquiring multiple feature points whose three-dimensional coordinate positions have been determined, i.e., point cloud data. In addition to the three-dimensional coordinate position, the feature points may also be associated with pixel values ​​indicating color (R value, G value, B value), grayscale color (256 levels), or black and white color (2 levels).

[0049] Furthermore, the point cloud data acquisition unit 132 may use MVS (Multi-View Stereo), a technique that uses multiple image data and point cloud data acquired from multiple image data to reconstruct denser point cloud data, by reconstructing denser point cloud data from the point cloud data acquired by SfM and multiple image data.

[0050] The division unit 133 divides the point cloud data acquired by the point cloud data acquisition unit 132 into partial point cloud data, each representing a different object. For example, the division unit 133 executes the division model 121 generated by the learning unit 131 and stored in the storage unit 12, inputs the point cloud data acquired by the point cloud data acquisition unit 132 to the division model 121, and acquires one or more partial point cloud data from the division model 121, thereby dividing the point cloud data into multiple partial point cloud data.

[0051] The position relationship identification unit 134 identifies the positional relationship between the object indicated by the partial point cloud data and other objects for each of the multiple partial point cloud data divided by the division unit 133. The position relationship identification unit 134 identifies the positional relationship between the object indicated by the partial point cloud data and other objects as one of the following: Disjoin, Inside, Meet, CoveredBy, Overlap, or Contains.

[0052] For example, the position relationship identification unit 134 executes the position relationship identification model 122 generated by the learning unit 131 and stored in the storage unit 12, inputs each of the multiple partial point cloud data divided by the division unit 133 to the position relationship identification model 122, and obtains the position relationship information output from the position relationship identification model 122 to identify the position relationship corresponding to the object corresponding to each of the multiple partial point cloud data.

[0053] The recognition unit 135 recognizes an object corresponding to each of the multiple partial point cloud data sets divided by the division unit 133, based on the positional relationship identified by the positional relationship identification unit 134 for each of the multiple partial point cloud data sets. The recognition unit 135 inputs the partial point cloud data sets divided by the division unit 133 and positional relationship information indicating the positional relationship identified by the positional relationship identification unit 134 for each of the partial point cloud data sets to the object recognition model 123 generated by the learning unit 131 and stored in the storage unit 12, and recognizes the object corresponding to the partial point cloud data by obtaining a label (e.g., object name) output from the object recognition model 123.

[0054] For example, the recognition unit 135 selects one subpoint cloud data from among multiple subpoint cloud data. The recognition unit 135 inputs a set of the selected subpoint cloud data and positional relationship information indicating the positional relationship identified with that subpoint cloud data to the object recognition model 123, thereby obtaining a label corresponding to that subpoint cloud data from the object recognition model 123. The recognition unit 135 selects one unselected subpoint cloud data from among multiple subpoint cloud data, inputs a set of that subpoint cloud data and positional relationship information corresponding to that subpoint cloud data to the object recognition model 123, and repeats the process of obtaining a label, thereby recognizing an object corresponding to multiple subpoint cloud data.

[0055] Furthermore, if the positional relationship of the recognized object with other objects is one of Inside, Meet, Covered By, Overlap, or Contains, the recognition unit 135 may compare the point cloud data of a predetermined three-dimensional model corresponding to the object with the partial point cloud data of the object and supplement the partial point cloud data with points that could not be obtained by other objects.

[0056] The attribute identification unit 136 identifies at least one of the following attributes of the object based on the partial point cloud data corresponding to the object recognized by the recognition unit 135: the type of object, the position of the object, the direction the object is facing, the shape of the object, the area of ​​the object, the volume of the object, and the state of the object. For example, the attribute identification unit 136 estimates the surface of the object based on the partial point cloud data corresponding to the object recognized by the recognition unit 135 and generates a 3D model of the object. The attribute identification unit 136 identifies the surface area, volume, and geometric shape of the generated 3D model, as well as the position of the object and the direction in which the front of the object is facing.

[0057] Furthermore, the memory unit 12 stores an attribute identification model that has been pre-trained using sets of partial point cloud data, object type, and object state as training data. The attribute identification unit 136 receives partial point cloud data of the recognized object as input to the attribute identification model and identifies the object type and state by outputting information indicating the object type and state from the attribute identification model.

[0058] The memory control unit 137 stores the attributes of the object identified by the attribute identification unit 136 in the memory unit 12. For example, the memory control unit 137 generates an object ID to identify the object recognized by the recognition unit 135. The memory control unit 137 associates the generated object ID with the date on which the recognition unit 135 recognized the object, the label acquired by the recognition unit 135, and attribute information indicating the attributes of the object identified by the attribute identification unit 136, and stores this as object information in the object DB 124 of the memory unit 12.

[0059] [Detection of changes in object state] Next, the process by which the control unit 13 detects changes in the state of an object will be described. The point cloud data acquisition unit 132, division unit 133, position relationship identification unit 134, recognition unit 135, attribute identification unit 136, memory control unit 137, detection unit 138, and notification unit 139 of the control unit 13 work together to detect changes in the state of the recognized object and notify the facility user, etc. The following describes in detail the functions of the point cloud data acquisition unit 132, division unit 133, position relationship identification unit 134, recognition unit 135, attribute identification unit 136, memory control unit 137, detection unit 138, and notification unit 139 related to the detection of changes in the state of an object.

[0060] First, the point cloud data acquisition unit 132 acquires point cloud data for multiple different time points. For example, if the system detects changes in the state of an object daily, the point cloud data acquisition unit 132 acquires video data from the imaging device 2 daily. The point cloud data acquisition unit 132 acquires point cloud data from the video data at the time the video data is acquired. In this way, the point cloud data acquisition unit 132 acquires point cloud data corresponding to each of the video data acquired daily.

[0061] Next, the splitting unit 133 divides the point cloud data acquired by the point cloud data acquisition unit 132 for each of several different time points into multiple partial point cloud data. For example, the splitting unit 133 inputs the point cloud data into the splitting model 121 in response to the point cloud data acquisition unit 132 acquiring the point cloud data, and then acquires multiple partial point cloud data from the splitting model 121, thereby dividing the point cloud data into multiple partial point cloud data.

[0062] Next, the position relationship identification unit 134 identifies the position relationship for each of the multiple partial point cloud data that have been divided for each of the multiple different time points. For example, in response to the division unit 133 dividing the point cloud data into multiple partial point cloud data, the position relationship identification unit 134 inputs each of the multiple partial point cloud data into the position relationship identification model 122 and identifies the position relationship by obtaining position relationship information from the position relationship identification model 122.

[0063] Next, the recognition unit 135 identifies an object corresponding to each of the multiple subpoint cloud data corresponding to multiple different time periods, based on each of the multiple subpoint cloud data corresponding to multiple different time periods and the positional relationship identified by the positional relationship identification unit 134 for the said subpoint cloud data corresponding to multiple different time periods. For example, in response to the positional relationship identification unit 134 identifying the positional relationship for each of the multiple subpoint cloud data, the recognition unit 135 inputs the said subpoint cloud data and said positional relationship into the object recognition model 123, and recognizes the object by obtaining the object label from the object recognition model 123.

[0064] Next, the attribute identification unit 136 identifies the attributes of the object recognized by the recognition unit 135 in accordance with multiple different time periods. For example, in response to the recognition unit 135 recognizing an object, the attribute identification unit 136 identifies the attributes of the object based on the corresponding partial point cloud data. The storage control unit 137 associates the object ID of the recognized object, the date on which the recognition unit 135 recognized the object, the label acquired by the recognition unit 135, and the attribute information indicating the attributes of the object identified by the attribute identification unit 136, and stores this as object information in the object DB 124 of the storage unit 12. As a result, object information corresponding to multiple different time periods is stored in the object DB 124.

[0065] The detection unit 138 detects changes in the attributes of an object based on the attributes of the object identified by the attribute identification unit 136 in response to multiple different times. For example, the detection unit 138 refers to object information corresponding to each of multiple different times stored in the object DB 124 and identifies the same object detected at each of the multiple different times. For example, the detection unit 138 identifies the same object detected at each of the multiple different times based on the object name indicated by the object information, the installation location, shape, length, and volume matching rate included in the attribute information. The detection unit 138 detects that the attributes of an object have changed if the state of the object included in the object information of the object detected at the most recent time differs from the state of the object included in the object information of the object detected at the immediately preceding time.

[0066] The detection unit 138 may also refer to object information corresponding to multiple different time points and detect when a new object has been recognized or when a previously recognized object is no longer recognized.

[0067] The notification unit 139 notifies the terminal 3 of changes in the attributes of an object detected by the detection unit 138. For example, when the detection unit 138 detects that the attributes of an object have changed, the notification unit 139 notifies the terminal 3 of the latest object information corresponding to that object. In addition, if the detection unit 138 detects that a new object has been recognized or that a previously recognized object is no longer recognized, the notification unit 139 may also notify the terminal 3 of the latest object information for that object.

[0068] [Operation Flow] Next, the processing flow of the information processing device 1 will be explained. Figure 5 is a flowchart showing the processing flow of the information processing device 1 in detecting changes in the state of an object. It should be assumed that, prior to the processing shown in Figure 5, the division model 121, the positional relationship identification model 122, and the object recognition model 123 are generated and stored in the storage unit 12, and that object information of previously recognized objects is also stored in the storage unit 12.

[0069] First, the point cloud data acquisition unit 132 acquires video data from the imaging device 2, and then acquires point cloud data based on that video data (S1). Next, the splitting unit 133 acquires multiple point cloud data by splitting the point cloud data acquired by the point cloud data acquisition unit 132 into multiple partial point cloud data (S2).

[0070] Next, the positional relationship identification unit 134 identifies the positional relationship for each of the divided subpoint cloud data (S3). Next, the recognition unit 135 recognizes an object by identifying an object label based on the divided subpoint cloud data and the positional relationship identified for each of the divided subpoint cloud data (S4).

[0071] Next, the attribute identification unit 136 identifies the attributes of the recognized object based on the partial point cloud data corresponding to the object (S5). Next, the memory control unit 137 associates the object ID of the recognized object, the date on which the recognition unit 135 recognized the object, the label identified by the recognition unit 135, and the attribute information indicating the attributes of the object identified by the attribute identification unit 136, and stores this as object information in the object DB 124 (S6).

[0072] Next, the detection unit 138 refers to the object DB 124 and determines whether or not there are any objects whose state has changed, based on the attributes of the objects identified by the attribute identification unit 136 in response to multiple different time points (S7). If the detection unit 138 determines that there are objects whose state has changed (YES in S7), it notifies the terminal 3 of the latest object information corresponding to the objects whose state has changed (S8). If the detection unit 138 determines that there are no objects whose state has changed (NO in S7), it terminates the process related to this flowchart.

[0073] [Differentiation] In the above-described embodiment, the position relationship identification unit 134 identifies, but is not limited to, one of the following position relationships for the partial point cloud data: Disjoin, Inside, Meet, CoveredBy, Overlap, or Contains, as the position relationship between the object indicated by the partial point cloud data and other objects. The position relationship identification unit 134 may also identify the density of the point cloud corresponding to the partial point cloud data as the position relationship between the object indicated by the partial point cloud data and other objects.

[0074] Objects that are relatively close to the imaging device 2 will have a larger area of ​​the object included in the image data compared to objects that are relatively far from the imaging device 2, resulting in a larger number of feature points. Therefore, the point density of the partial point cloud data corresponding to objects that are relatively close to the imaging device 2 will be higher than the point density of the partial point cloud data corresponding to objects that are relatively far from the imaging device 2. Consequently, the information processing device 1 will determine that the point density corresponding to the partial point cloud data indicates the positional relationship between the object represented by the partial point cloud data and other objects.

[0075] In this case, the object recognition model 123 may output a label for identifying the recognized object when it receives partial point cloud data and density information of the point cloud indicating the positional relationship corresponding to the partial point cloud data. The recognition unit 135 may recognize an object by inputting the partial point cloud data and density information to the object recognition model 123 and obtaining a label for identifying the recognized object. By doing so, the recognition unit 135 can identify Disjoin, Inside, Meet, CoveredBy, Overlap, and Contains as positional relationships and improve the accuracy of object recognition, similar to when recognizing an object based on these positional relationships.

[0076] Furthermore, Disjoin, Inside, Meet, CoveredBy, Overlap, and Contains, which indicate the positional relationship, may be used as first positional relationship information, and density information, which indicates the density of the point cloud corresponding to the partial point cloud data, may be used as second positional relationship information. The object recognition model 123 may output a label for identifying the recognized object when it receives partial point cloud data and the first and second positional relationship information indicating the positional relationship corresponding to the partial point cloud data as input. By recognizing objects using such an object recognition model 123, the accuracy of object recognition can be further improved.

[0077] Furthermore, in the above-described embodiment, the information processing device 1 uses a division model 121 and an object recognition model 123 to acquire partial point cloud data and recognize objects, but is not limited to this. If the types of objects to be divided and the labels of the objects to be recognized are predetermined, the information processing device 1 may use k-means clustering to acquire partial point cloud data and recognize objects.

[0078] [Effects of Information Processing Device 1] As described above, the information processing device 1 according to this embodiment acquires point cloud data containing multiple three-dimensional coordinates corresponding to each of multiple objects, and divides the acquired point cloud data into partial point cloud data, which are point cloud data representing each of the multiple objects. For each of the divided partial point cloud data, the information processing device 1 identifies the positional relationship between the object represented by the partial point cloud data and other objects, and recognizes the object corresponding to each of the divided partial point cloud data based on the positional relationship identified for each of the divided partial point cloud data. In this way, the information processing device 1 can improve the accuracy of object recognition.

[0079] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of its gist. For example, all or part of the apparatus can be configured by functionally or physically distributing and integrating in any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combinations are combined with the effects of the original embodiments. [Explanation of Symbols]

[0080] 1. Information Processing Device 2. Imaging device 3 terminals 11 Communications Department 12 Storage section 13 Control Unit 121-part model 122 Positional Relationship Identification Model 123 Object Recognition Models 124 Object DB 131 Learning Department 132 Point cloud data acquisition unit 133 Split part 134 Positional relationship identification unit 135 Recognition part 136 Attribute identification part 137 Memory Control Unit 138 Detection unit 139 Notification Department

Claims

1. An acquisition unit that acquires point cloud data containing multiple three-dimensional coordinates corresponding to multiple objects, A division unit divides the point cloud data acquired by the acquisition unit into partial point cloud data, which are point cloud data representing each of multiple objects, based on the point cloud data acquired by the acquisition unit. For each of the multiple partial point cloud data divided by the division unit, a positional relationship identification unit identifies the positional relationship between the object represented by the partial point cloud data and an object different from that object. A recognition unit recognizes an object corresponding to each of the multiple partial point cloud data, based on each of the multiple partial point cloud data divided by the division unit and the positional relationship identified by the positional relationship identification unit for the said partial point cloud data. An information processing device having

2. The position relationship identification unit identifies the position relationship corresponding to the object corresponding to the partial point cloud data by inputting the partial point cloud data divided by the division unit into a position relationship identification model that outputs position relationship information indicating the position relationship when the partial point cloud data is input, and by acquiring the position relationship information output from the position relationship identification model. The information processing apparatus according to claim 1.

3. The system further includes a learning unit that generates the positional relationship identification model by learning partial point cloud data corresponding to objects whose positional relationships have been identified and positional relationship information indicating said positional relationships as training data. The position relationship identification unit inputs the partial point cloud data divided by the division unit into the position relationship identification model generated by the learning unit, and acquires the position relationship information output from the position relationship identification model, thereby identifying the position relationship corresponding to the object indicated by the partial point cloud data. The information processing apparatus according to claim 2.

4. When the recognition unit receives the partial point cloud data and positional relationship information indicating the positional relationship as input, it receives the partial point cloud data divided by the division unit and the positional relationship information indicating the positional relationship identified by the positional relationship identification unit for the partial point cloud data as input to an object recognition model that outputs a label for identifying the recognized object, and acquires the label output from the object recognition model to recognize the object corresponding to the partial point cloud data. The information processing apparatus according to claim 1.

5. The system further includes a learning unit that generates the object recognition model by learning the partial point cloud data corresponding to a previously recognized object, the positional relationship corresponding to the object, and the label corresponding to the object as training data. The recognition unit inputs the partial point cloud data divided by the division unit and the positional relationship information indicating the positional relationship identified by the positional relationship identification unit for the partial point cloud data into the object recognition model generated by the learning unit, and recognizes the object corresponding to the partial point cloud data by obtaining the label output from the object recognition model. The information processing apparatus according to claim 4.

6. The system further includes a learning unit that generates a segmented model that outputs multiple subpoint cloud data in response to point cloud data input, by learning from point cloud data corresponding to multiple pre-recognized objects and multiple subpoint cloud data corresponding to each of those objects as training data. The division unit inputs the point cloud data acquired by the acquisition unit to the division model generated by the learning unit, and acquires one or more partial point cloud data from the division model, thereby dividing the point cloud data into multiple partial point cloud data. The information processing apparatus according to claim 1.

7. The positional relationship identification unit identifies the positional relationship as one of the following: a positional relationship in which the object does not overlap with the other object; a positional relationship in which at least a part of the object is concealed by the other object being adjacent to the object; a positional relationship in which the other object overlaps the top of the object; a positional relationship in which a part of the object is covered by the other object; or a positional relationship in which the object contains the other object. The information processing apparatus according to claim 1.

8. The positional relationship identification unit identifies the density of the point cloud corresponding to the partial point cloud data as the positional relationship. The information processing apparatus according to claim 1.

9. An attribute identification unit identifies at least one of the following as attributes of the object based on the partial point cloud data corresponding to the object recognized by the recognition unit: the type of the object, the position of the object, the direction the object is facing, the shape of the object, the area of ​​the object, the volume of the object, and the state of the object. A storage control unit that causes the attribute identification unit to store the attributes of the object identified by the attribute identification unit in a storage unit, It further possesses, The information processing apparatus according to claim 1.

10. The acquisition unit acquires the point cloud data for each of several different time points, The division unit divides the point cloud data acquired by the acquisition unit for each of the multiple different time points into multiple partial point cloud data. The positional relationship identification unit identifies the positional relationship for each of the multiple partial point cloud data divided for each of the multiple different time points, The recognition unit recognizes an object corresponding to each of the multiple partial point cloud data corresponding to each of the multiple partial point cloud data corresponding to each of the multiple different time periods, based on each of the multiple partial point cloud data corresponding to each of the multiple different time periods and the positional relationship identified by the positional relationship identification unit for the said partial point cloud data corresponding to each of the multiple different time periods. The attribute identification unit identifies the attributes of the object recognized by the recognition unit in accordance with the plurality of different time periods, A detection unit detects changes in the attributes of the object based on the attributes of the object identified by the attribute identification unit in response to the aforementioned multiple different time periods, A notification unit that notifies of changes in the attributes of the object detected by the detection unit, It further possesses, The information processing apparatus according to claim 9.

11. A computer executes The steps include obtaining point cloud data containing multiple three-dimensional coordinates corresponding to each of multiple objects, Based on the acquired point cloud data, the step of dividing the point cloud data into partial point cloud data, which are point cloud data representing each of multiple objects, For each of the divided subpoint cloud data, the step is to identify the positional relationship between the object represented by the subpoint cloud data and other objects, A step of recognizing an object corresponding to each of the multiple divided partial point cloud data, based on each of the multiple divided partial point cloud data and the positional relationship identified with respect to the partial point cloud data, An information processing method having

12. Computers, An acquisition unit that acquires point cloud data containing multiple three-dimensional coordinates corresponding to each of multiple objects. A division unit divides the point cloud data acquired by the acquisition unit into partial point cloud data, which are point cloud data representing each of multiple objects. For each of the multiple partial point cloud data divided by the division unit, a positional relationship identification unit identifies the positional relationship between the object indicated by the partial point cloud data and an object different from that object, and A recognition unit recognizes an object corresponding to each of the multiple partial point cloud data, based on each of the multiple partial point cloud data divided by the division unit and the positional relationship identified by the positional relationship identification unit for each of the partial point cloud data. A program that makes it function as such.

Images