Program, information processing apparatus, method, and system

Abstract

To easily associate image data with point group data.SOLUTION: A program for operating a computer including a processor, a memory, a photographing unit configured to photograph an image by visible light, and a three dimensional sensor having the same orientation direction as the photographing unit. The program causes a processor to execute a step of receiving a photographing instruction of an image, a step of acquiring image data from a photographing unit based on the received photographing instruction, a step of acquiring point cloud data from a three dimensional sensor based on the received photographing instruction, a step of associating a target image of a target object included in the image data with point data of the target object included in the point cloud data, and a step of incorporating the point cloud data in which the point data is associated with the target image into metadata of the image data and storing the image data in which the point cloud data is incorporated into the metadata.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a program, an information processing device, a method, and a system. [Background technology]

[0002] Conventionally, techniques for associating detection results from a camera with detection results from a three-dimensional sensor such as a LiDAR (Laser Imaging Detection and Ranging) device have been known. For example, Patent Document 1 discloses a technique for integrating detection results from a camera and LiDAR. The technique disclosed in Patent Document 1 acquires first data indicating information about a first object based on image data generated by a camera, and acquires second data indicating information about a second object based on point cloud data generated by the LiDAR. The first data includes information about the length of the first object, and the second data includes information about the length of the second object. The technique disclosed in Patent Document 1 generates integrated data indicating information about objects in a measurement area by integrating the first data and second data, where the information about the length of the first object and the information about the length of the second object match. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2024-017636 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology disclosed in Patent Document 1 requires calculating the coordinates of the first object when acquiring the first data, and then calculating information about the length of the first object based on these coordinates. Furthermore, the technology disclosed in Patent Document 1 also requires comparing information about the length of the first object with information about the length of the second object to determine whether they match before generating integrated data. This creates a problem in that the integration process between image data generated by a camera and point cloud data generated by LiDAR becomes complicated.

[0005] An object of the present disclosure is to easily associate image data with point cloud data. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, a program according to one aspect of the present disclosure is a program for operating a computer including a processor, a memory, an image capturing unit that captures an image using visible light, and a 3D sensor having the same orientation direction as the image capturing unit. The program causes the processor to execute the following steps: accepting an instruction to capture an image; acquiring image data from the image capturing unit based on the accepted instruction; acquiring point cloud data from the 3D sensor based on the accepted instruction; associating a target image of an object included in the image data with point data of the object included in the point cloud data; incorporating the point cloud data, in which the point data is associated with the target image, into metadata of the image data; and storing the image data with the point cloud data incorporated into the metadata. [Effects of the Invention]

[0007] According to the present disclosure, image data and point cloud data can be easily associated with each other. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of the overall configuration of a system 1. FIG. [Figure 2]2 is a block diagram showing an example of the configuration of a terminal device 10 shown in FIG. 1. FIG. [Figure 3] 2 is a block diagram showing an example of a functional configuration of a server 20 shown in FIG. 1. FIG. [Figure 4] FIG. 4 is a diagram showing the data structure of an image data table 2021 shown in FIG. 3. [Figure 5] 10 is a flowchart showing an example of the operation of the terminal device 10 when presenting an embedded image to a user. [Figure 6] FIG. 10 is a schematic diagram showing an example of the display screen of the display 141 when an embedded image is presented to the user. [Figure 7] FIG. 10 is a schematic diagram showing an example of the display screen of the display 141 when the size of an object is presented to the user. [Figure 8] FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all drawings describing the embodiments, common components are designated by the same reference numerals, and repeated description will be omitted. Note that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Furthermore, not all components shown in the embodiments are necessarily essential components of the present disclosure. Furthermore, each drawing is a schematic diagram and is not necessarily a precise illustration.

[0010] [1. Overview] The system according to this embodiment acquires image data of an object from a camera and acquires point cloud data from a 3D sensor. The system according to this embodiment associates an image of the object (object image) contained in the image data with point data of the object contained in the point cloud data. The system according to this embodiment stores the image data with the point cloud data incorporated into the metadata of the image data.

[0011] [2. Overall system configuration diagram] Fig. 1 is a block diagram showing an example of the overall configuration of a system 1. The system 1 shown in Fig. 1 includes, for example, a terminal device 10 and a server 20. The terminal device 10 and the server 20 are connected for communication via, for example, a network 80. Note that the server 20 is not an essential component of the present disclosure, and one aspect of the present disclosure may be realized by the terminal device 10 alone. In this case, the terminal device 10 performs, for example, data management and data processing instead of the server 20.

[0012] 1 shows an example in which the system 1 includes one terminal device 10, but the number of terminal devices 10 included in the system 1 is not limited to one. The number of terminal devices 10 included in the system 1 may be, for example, two or more.

[0013] 1 shows an example in which the server 20 is configured with one information processing device, but the server 20 may be configured as a collection of multiple information processing devices. The way in which multiple functions required to realize the server 20 according to this embodiment are allocated to one or multiple pieces of hardware can be determined appropriately in consideration of the processing capacity of each piece of hardware and / or the specifications required for the server 20.

[0014] 1 is an information processing device operated by a user who takes a photograph of an object. The terminal device 10 may be, for example, a mobile terminal such as a smartphone or a tablet, or may be a stationary personal computer (PC) or a laptop PC. Furthermore, for example, the terminal device 10 may be a wearable terminal such as a head mounted display (HMD) or a wristwatch terminal.

[0015] The terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19. The input device 13 is a device for receiving input operations from a user (for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.). The output device 14 is a device for presenting information to a user (a display, a speaker, etc.).

[0016] The server 20 is, for example, an information processing device realized by a computer connected to the network 80, and manages embedded image data (details will be described later). As shown in Fig. 1, the server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29. The input / output IF 23 functions as an input device for receiving input operations from a user, and as an interface with an output device for outputting information to the user.

[0017] Each information processing device is configured by a computer 90 (see FIG. 8) equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer 90 and the basic functional configuration of the computer 90 realized by the hardware configuration will be described later. For each of the terminal device 10 and the server 20, descriptions that overlap with the basic hardware configuration and basic functional configuration of a computer that will be described later will be omitted.

[0018] <2.1 Terminal device configuration> Fig. 2 is a block diagram showing an example configuration of the terminal device 10 shown in Fig. 1. As shown in Fig. 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 170, a microphone 171, a speaker 172, a camera 160, a position information sensor 150, an acceleration sensor 155, a three-dimensional sensor 165, a storage unit 180, and a control unit 190. The blocks included in the terminal device 10 are electrically connected by, for example, a bus or the like.

[0019] The communication unit 120 performs processing such as modulation and demodulation for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on signals generated by the control unit 190 and transmits the signals to the outside. The communication unit 120 performs reception processing on signals received from the outside and outputs the signals to the control unit 190.

[0020] The input device 13 is a device for inputting instructions or information by a user operating the terminal device 10. The input device 13 may be realized by, for example, a keyboard, a mouse, a reader, etc. If the terminal device 10 is a mobile terminal or the like, the input device 13 may be realized by, for example, a touch-sensitive device 131, which inputs instructions by touching the operation surface. The input device 13 converts instructions input by the user into electrical signals and outputs the electrical signals to the control unit 190. The input device 13 may also include, for example, a receiving port that receives electrical signals input from an external input device.

[0021] The output device 14 is a device for presenting information to a user operating the terminal device 10. The output device 14 is realized, for example, by a display 141 or the like. The display 141 displays data according to the control of the control unit 190. The display 141 is realized, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display or the like.

[0022] The audio processing unit 170 performs, for example, digital-to-analog conversion processing of an audio signal. The audio processing unit 170 converts a signal provided from the microphone 171 into a digital signal and provides the converted signal to the control unit 190. The audio processing unit 170 also provides the audio signal to the speaker 172. The audio processing unit 170 is realized, for example, by a processor for audio processing. The microphone 171 receives audio input and provides an audio signal corresponding to the audio input to the audio processing unit 170. The speaker 172 converts the audio signal provided from the audio processing unit 170 into audio and outputs the audio to the outside of the terminal device 10.

[0023] The camera 160 is an example of an imaging unit according to an embodiment of the present disclosure, and captures images using visible light. That is, the camera 160 is a device that receives visible light using a light-receiving element and outputs image data as an imaging signal. The camera 160 captures an image of a subject in a certain direction and within a certain imaging range relative to the terminal device 10, and outputs image data as the imaging result. If the camera 160 has a function that allows the imaging range, or more precisely, the angle of view, to be adjustable, the camera 160 also outputs information regarding this angle of view. This function is called a zoom function. For example, the camera 160 is fixed to the housing of the terminal device 10, so that the imaging direction of the camera 160 relative to the terminal device 10, i.e., the orientation direction of the camera 160, is set to a certain direction.

[0024] The position information sensor 150 is a sensor that detects the position of the terminal device 10 and is generally a GNSS device, such as a GPS module. The GPS module is a receiving device used in a satellite positioning system. In a satellite positioning system, signals are received from at least three or four satellites, and the current position of the terminal device 10 equipped with the GPS module is detected as coordinate values ​​based on the received signals. The position information sensor 150 may detect the current position of the terminal device 10 from the position of a wireless base station to which the terminal device 10 connects via the communication unit 120.

[0025] The acceleration sensor 155 is a sensor that detects the acceleration applied to the terminal device 10. Preferably, the acceleration sensor 155 has a function of detecting the tilt around each axis (X-axis, Y-axis, Z-axis) of a three-dimensional coordinate system with the position of the terminal device 10 as the origin. The acceleration sensor 155 having such a function can detect the attitude of the terminal device 10, that is, the direction with respect to the X-axis, Y-axis, and Z-axis, by detecting the gravitational acceleration of the force of gravity with respect to the Earth. Furthermore, since the pointing direction of the camera 160 is set in a fixed direction, the pointing direction of the camera 160 in three-dimensional space can be obtained based on the detection result of the acceleration sensor 155.

[0026] The three-dimensional sensor 165 generates and outputs point cloud data of the measurement object. The three-dimensional sensor 165 has, for example, the same orientation as the camera 160. A known example of such a three-dimensional sensor 165 is a LiDAR. The LiDAR irradiates (scans) a wide range of the measurement object with a laser beam, receives the reflected light, and acquires three-dimensional position data of the surface of the measurement object based on the presence or absence of the reflected light and the timing of receiving the reflected light. The three-dimensional position data acquired by the LiDAR is three-dimensional point cloud data, which is the reflected position of the laser beam. In the following description, it is assumed that the three-dimensional sensor 165 is a LiDAR. In other words, in the present specification, point cloud data strictly refers to three-dimensional point cloud data.

[0027] The acquisition range (measurement range) of point cloud data by three-dimensional sensor 165 is set to include at least the imaging range of camera 160. If camera 160 has a zoom function, point cloud data can be acquired in a measurement range that includes at least the imaging range when the angle of view of camera 160 is at its maximum (i.e., closer to the wide-angle side). Alternatively, if the measurement range of three-dimensional sensor 165 is variable, the measurement range of three-dimensional sensor 165 can be set to match the imaging range of camera 160.

[0028] The type of 3D sensor 165 is not limited to LiDAR. Examples of 3D sensors 165 other than LiDAR include a structured light 3D scanner, a time-of-flight camera (ToF camera), a stereo camera, and an RGB-D camera.

[0029] Furthermore, three-dimensional sensor 165 does not have to directly generate point cloud data. For example, point cloud data may be generated by applying a specific process / algorithm to two-dimensional data (including depth information), RGB-D data, etc. that is directly generated by three-dimensional sensor 165. In other words, point cloud data may be obtained directly or indirectly from three-dimensional sensor 165.

[0030] The storage unit 180 is realized by, for example, the memory 15 and the storage 16, and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, an application program 181, image data 182, and point cloud data 183. The application program 181 includes, for example, a browser that is installed by default in the terminal device 10. Note that at least one of the image data 182 and the point cloud data 183 may be stored, for example, in the storage unit 202 described below.

[0031] Image data 182 is data obtained by camera 160 capturing an image of a target area, which is a subject. In other words, an image of the target area is an example of a "visible light image" according to one embodiment of the present disclosure. Hereinafter, an image of the target area will be simply referred to as an "image." Image data 182 includes, for example, metadata, pixel data, object information, compression information, color space information, resolution, etc.

[0032] Metadata is information about the image itself, including, for example, shooting information (e.g., shooting date and time, shooting location, shooting conditions, information about the equipment used, such as the camera), file information (e.g., file format, file size, color depth), and color space profiling. Pixel data is data that represents the pixel value, brightness, etc. of each pixel that makes up an image. The pixel value is also an index that reflects the reflectance and reflective properties of the surface of an object.

[0033] Object information is information about objects present in the object area, including, for example, the type, size, color, and number of objects. Compression information is information about the compression method and compression rate when an image file is compressed. Color space information is information indicating the color space in which the image is expressed (e.g., RGB color space, CMYK color space). Resolution is information about the width, height, and pixel density (e.g., DPI) of the image.

[0034] The point cloud data 183 is data obtained by measuring a target area that is a measurement target with the three-dimensional sensor 165. The point cloud data 183 includes, for example, metadata, coordinate data, size information, normal vectors, density information, reflection information, attribute information, and the like.

[0035] The metadata is information about the point cloud data 183 itself, and includes, for example, generation information (e.g., information about the method and device that generated the point cloud data), coordinate system information, time information, etc. The coordinate system information is information indicating which coordinate system the point cloud data 183 is based on or into which coordinate system it has been converted. Examples of coordinate systems include a geographic coordinate system (longitude, latitude, altitude) and a robot coordinate system (reference point, posture information). The time information is information indicating the point in time at which the point cloud data 183 was generated.

[0036] The coordinate data are (X, Y, Z) coordinate values ​​indicating the position in three-dimensional space of each point constituting the point cloud data 183. The normal vector is a vector indicating the normal direction at each point constituting the point cloud data 183. The size information is information about the size of an object present in the target area, and includes, for example, each dimension such as length, width, height, inner diameter, outer diameter, and thickness, surface area, and volume. For example, the height of an object included in the size information is a value obtained by subtracting the Z coordinate value of the point with the lowest position in the point data from the Z coordinate value of the point with the highest position in the point data. The point data is point cloud data indicating the object, and is included in the point cloud data 183. Here, the size information also includes size information of the object.

[0037] The density information is information indicating the number of points existing around each point constituting the point cloud data 183, density distribution, etc., and is an index indicating the surface characteristics of the object (e.g., complexity of the surface shape, surface curvature, edge position). The reflection information is information regarding the reflection intensity, brightness, etc. of each point constituting the point cloud data 183. The attribute information is information regarding the attributes of each point constituting the point cloud data 183, and includes, for example, point classification (e.g., ground point, building point) and segmentation information which is information about the object (object) to which the point belongs.

[0038] Here, the target area that is the subject of camera 160 is, for example, the same as the target area that is the measurement target of three-dimensional sensor 165. Also, it is assumed that a specific object that serves as the target exists in the target area. Furthermore, the number of objects that exist in the target area may be one or two or more.

[0039] The control unit 190 is realized by the processor 19 reading an application program 181 stored in the storage unit 180 and executing instructions included in the application program 181. The control unit 190 controls the operation of the terminal device 10. The control unit 190 operates in accordance with the program to fulfill the functions of an operation reception unit 191, a transmission / reception unit 192, a presentation control unit 193, a camera control unit 194, a sensor control unit 195, and a data generation unit 196.

[0040] The operation reception unit 191 performs processing for receiving instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives information based on instructions input from a keyboard, a mouse, or the like.

[0041] Furthermore, the operation reception unit 191 receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives a voice signal that is input from the microphone 171 and converted into a digital signal by the voice processing unit 170. For example, the operation reception unit 191 analyzes the received voice signal and extracts a predetermined noun, thereby acquiring an instruction from the user.

[0042] The transmitting / receiving unit 192 performs processing for the terminal device 10 to transmit and receive data to and from external devices in accordance with a communication protocol. Specifically, for example, the transmitting / receiving unit 192 transmits content input by a user to the external device. The transmitting / receiving unit 192 also receives information about the user from the external device.

[0043] The presentation control unit 193 controls the output device 14 to present to the user information generated by each functional unit of the terminal device 10. Specifically, for example, the presentation control unit 193 causes the display 141 to display the information generated by each functional unit of the terminal device 10. In addition, the presentation control unit 193 causes the speaker 172 to output the information generated by each functional unit of the terminal device 10.

[0044] The camera control unit 194 performs overall control of the camera 160. For example, while the system 1 is operating (while the application program 181 is running), the camera control unit 194 operates the camera 160 based on a user operation and causes the camera 160 to capture an image of a target area. For example, the camera control unit 194 acquires image data 182 output from the camera 160 and stores the image data in the storage unit 180.

[0045] Furthermore, for example, the camera control unit 194 acquires the measurement values ​​(coordinate values) of the position information sensor 150 and the measurement values ​​of the acceleration sensor 155 at the time when the camera 160 captured the image of the target area. The camera control unit 194 stores the acquired measurement values ​​in the storage unit 180 as metadata of the image data 182, for example.

[0046] The sensor control unit 195 performs overall control of the three-dimensional sensor 165. For example, during operation of the system 1, the sensor control unit 195 operates the three-dimensional sensor 165 based on a user operation and causes the three-dimensional sensor 165 to generate point cloud data 183 of the target area. For example, the sensor control unit 195 acquires the point cloud data 183 from the three-dimensional sensor 165 at the same time that the camera control unit 194 acquires image data 182 from the camera 160. For example, the sensor control unit 195 acquires the point cloud data 183 output from the three-dimensional sensor 165 and stores it in the storage unit 180.

[0047] The data generation unit 196 associates the target image included in the image data 182 with the point data of the target object included in the point cloud data 183. The target image is an image portion that shows the target object and is included in the image indicated by the image data 182. Specifically, for example, the data generation unit 196 associates each pixel that makes up the target image in the image indicated by the image data 182 with each point that makes up the point data in the point cloud data 183. If there are multiple targets in the target area, the data generation unit 196 performs this association as many times as the number of targets.

[0048] There are no particular limitations on the method for associating the target image with the point data, and various methods can be adopted. In this embodiment, the data generation unit 196 associates the target image with the point data based on the coordinate values ​​and pixel values ​​included in the image data 182 and the coordinate values ​​and density included in the point cloud data 183.

[0049] Specifically, for example, the data generation unit 196 reads the coordinate values ​​and pixel values ​​of each pixel that constitutes the target image from the metadata of the image data 182, and reads the coordinate values ​​and density information of each point that constitutes the point data from the point cloud data 183. Then, the data generation unit 196 associates each pixel that constitutes the target image with each point that constitutes the point data, for example, while comparing the coordinate values ​​and pixel values ​​of the pixels with the coordinate values ​​and density information of the points. More specifically, the data generation unit 196 associates a collection of pixels and a collection of points that have the same or nearly the same coordinate values ​​and have a high correlation between pixel values ​​and density information.

[0050] Here, the data generation unit 196 determines the "relationship between pixel values ​​and density information" as follows, for example. That is, if the surface of the object is rough, various brightnesses and shadows appear in the pixel values, and the brightness indicated by the pixel values ​​varies greatly. In this case, the density indicated by the density information increases. On the other hand, if the surface of the object is smooth and approximately uniform, the pixel values ​​will also be approximately uniform, and the density indicated by the density information will also be approximately uniform. The data generation unit 196 utilizes this tendency to determine that there is a high correlation between the pixel values ​​and density information between a collection of pixels whose pixel values ​​indicate a large variation in brightness and a collection of points whose density indicated by the density information is high. Furthermore, the data generation unit 196 determines that there is a high correlation between the pixel values ​​and density information between a collection of pixels whose pixel values ​​are approximately uniform and a collection of points whose density indicated by the density information is approximately uniform.

[0051] The data generation unit 196 may associate the target image with the point data, for example, by analyzing the image data 182 to extract a target image. Specifically, for example, the data generation unit 196 may analyze the surface shape of the target object shown in the image indicated by the image data 182 and extract the surface shape of the target object shown in the target image. Then, the data generation unit 196 may associate the target image with the point data, for example, by associating the extracted surface shape with the shape of a virtual surface of the point data. The virtual surface is a surface virtually possessed by the point data, and is a surface with a shape and size corresponding to the surface of the target object. In other words, the virtual surface is a surface that virtually includes all points that make up the point data. The shape of the virtual surface may be determined, for example, based on the coordinate values ​​and density information of each point that makes up the point data, which are included in the point cloud data 183.

[0052] One example of this technique is object detection (object recognition) technology, which detects objects (subjects) in an image represented by image data, in artificial intelligence technology, particularly deep learning technology. Well-known object detection technologies include, for example, YOLO (You Only Look Once), R-CNN (Region Based Convolutional Neural Networks), and their advanced forms, Fast R-CNN, Faster R-CNN, and SSD (Single Shot MultiBox Detector).

[0053] Alternatively, for example, data generation unit 196 may employ a method for determining a transformation formula for projecting point data onto a target image, which is two-dimensional data. In other words, camera 160 and three-dimensional sensor 165 have the same orientation. Furthermore, the measurement range of the target area by three-dimensional sensor 165 includes at least the imaging range of the target area by camera 160, and sensor control unit 195 performs an operation for acquiring point cloud data 183 in response to an operation for acquiring image data 182 by camera control unit 194. Therefore, it is substantially determined that the image data 182 and the point cloud data 183 represent the same target object. Therefore, data generation unit 196 associates the target image with the point data by, for example, projecting the point data onto the target image and determining a transformation formula for this. Data generation unit 196 may, for example, temporarily store the determined transformation formula in storage unit 180.

[0054] The data generation unit 196 incorporates point cloud data 183, which associates point data with the target image, into the metadata of the image data 182. Specifically, for example, the data generation unit 196 performs the above-mentioned incorporating by adding reference information (path) of the point cloud data 183 to the metadata of the image data 182. This reference information (path) is included, for example, in the metadata of the point cloud data 183. Note that the data generation unit 196 may perform the above-mentioned incorporating by, for example, adding the point cloud data 183 itself to the metadata of the image data 182 instead of the reference information (path) of the point cloud data 183.

[0055] The data generation unit 196 stores the embedded image data. The embedded image data is image data 182 in a state in which the point cloud data 183 has been embedded in the metadata (reference information (path) of the point cloud data 183 has been added). Specifically, for example, the data generation unit 196 stores information related to the embedded image data in an image data table 2021. The image data table 2021 is a data table for managing information related to the embedded image data. Details of the image data table 2021 will be described later.

[0056] <2.2 Functional configuration of the server> 3 is a diagram showing an example of the functional configuration of the server 20. As shown in FIG. 3, the server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0057] The communication unit 201 performs processing for the server 20 to communicate with external devices. The storage unit 202 stores, for example, an image data table 2021. Note that the table stored in the storage unit 202 is not limited to the image data table 2021. For example, an image data table 2021 with different contents may be stored for each server that manages a different service. Furthermore, the image data table 2021 may be stored in a server related to a service for sharing information among multiple users.

[0058] The control unit 203 is realized by the processor 29 reading a program stored in the storage unit 202 and executing instructions included in the program. The program includes an application such as a web browser application. The program includes an interpreted programming language such as JavaScript (registered trademark) that is executed on a web browser application stored in the terminal device 10. The control unit 203 operates in accordance with the program to fulfill functions shown as a reception control module 2031 and a transmission control module 2032.

[0059] The reception control module 2031 controls the process of the server 20 receiving signals from external devices in accordance with a communication protocol. The transmission control module 2032 controls the process of the server 20 transmitting signals to external devices in accordance with a communication protocol.

[0060] [3 Data Structure] 4 is a diagram showing the data structure of tables stored in the server 20. Note that FIG. 4 is an example and does not exclude data that is not listed. Furthermore, even data listed in the same table may be stored in separate storage areas in the storage unit 202.

[0061] Fig. 4 is a diagram showing the data structure of the image data table 2021. The image data table 2021 shown in Fig. 4 is a table having columns of metadata, pixel data, and object information, with, for example, an image ID as a key. The information stored in the image data table 2021 is input and updated as appropriate by, for example, the data generation unit 196 via the reception control module 2031.

[0062] The "Image ID" item is an item that stores an identifier for uniquely identifying image data 182. The "Metadata" item is an item that stores metadata of the embedded image data. In the example of FIG. 4, the "Metadata" item includes at least the "Coordinate Data" item and the "Reference Information" item. The "Coordinate Data" item is an item that stores the coordinate values ​​(X, Y) of each pixel that constitutes the embedded image. The embedded image is an image indicated by the embedded image data. The "Reference Information" item is an item that stores reference information (path) of the point cloud data 183 that is added to the metadata of the embedded image data. The "Metadata" item may store other metadata, such as shooting information, for example.

[0063] The "pixel data" item is an item that stores the pixel values ​​of each pixel that makes up the embedded image, that is, the embedded image itself. Note that the image data table 2021 may have, for example, a column for the "image data reference information" item instead of the "pixel data" item. The "image data reference information" item is an item that stores reference information (path) for the embedded image. In this case, the embedded image is stored in a separate storage area in the storage unit 202, such as a file system or cloud storage. The "object information" item is an item that stores object information such as the type and size of the object.

[0064] It should be noted that it is not essential that the image data table 2021 be stored in the storage unit 202. The image data table 2021 may be stored, for example, in the storage unit 180, or may be stored in a storage area other than the storage unit 202 and the storage unit 180. In this case, for example, the server 20 becomes unnecessary.

[0065] [4 actions] The operation of the terminal device 10 when presenting an embedded image to the user will be described. Fig. 5 is a flowchart showing an example of the operation of the terminal device 10 when presenting an embedded image to the user. In the example of Fig. 5, it is assumed that the orientation directions of the camera 160 and the three-dimensional sensor 165 are each directed toward the target area in advance.

[0066] In step S11, the terminal device 10 accepts an instruction to capture an image (step of accepting an image capture instruction). Specifically, for example, the operation accepting unit 191 accepts a user instruction to instruct the camera 160 to capture an image of a target area. The operation accepting unit 191 transmits, for example, first acceptance information indicating that the above-mentioned user instruction has been accepted to the camera control unit 194 and the sensor control unit 195.

[0067] In step S12, the terminal device 10 acquires image data 182 from the camera 160 based on the accepted shooting instruction (step of acquiring image data). Specifically, for example, when the camera control unit 194 receives first reception information from the operation acceptance unit 191, it causes the camera 160 to capture an image of the target area. For example, the camera control unit 194 acquires image data 182 as an image capture signal output from the camera 160, and transmits it to the data generation unit 196. Also, for example, the camera control unit 194 stores the acquired image data 182 in the storage unit 180.

[0068] In step S13, terminal device 10 acquires point cloud data 183 from three-dimensional sensor 165 based on the accepted shooting instruction (step of acquiring point cloud data). Specifically, for example, upon receiving first acceptance information transmitted from operation acceptance unit 191, sensor control unit 195 causes three-dimensional sensor 165 to measure the target area and generate point cloud data 183. For example, sensor control unit 195 acquires point cloud data 183 output from three-dimensional sensor 165 and transmits it to data generation unit 196. Also, for example, sensor control unit 195 stores the acquired point cloud data 183 in storage unit 180.

[0069] The order of the processes of step S12 and step S13 is not limited to the example of Fig. 5. For example, the terminal device 10 may execute the process of step S12 and the process of step S13 simultaneously, or may execute the process of step S12 after the process of step S13.

[0070] In step S14, the terminal device 10 associates the target image included in the image data 182 with the point data included in the point cloud data 183 (association step). Specifically, for example, the data generation unit 196 extracts the target image from the image data 182 received from the camera control unit 194, and extracts point data from the point cloud data 183 received from the sensor control unit 195. The data generation unit 196 associates, for example, each pixel constituting the extracted target image with each point constituting the extracted point data. The specific method of association by the data generation unit 196 is as described above.

[0071] In step S15, the terminal device 10 generates embedded image data by incorporating point cloud data 183, which associates point data with the target image, into the metadata of the image data 182, and stores the generated embedded image data (storing step). Specifically, for example, the data generation unit 196 reads out reference information (path) of the point cloud data 183 from the point cloud data 183 received from the sensor control unit 195. The data generation unit 196 adds the read out reference information (path) to the metadata of the image data 182 received from the camera control unit 194, for example, to generate embedded image data.

[0072] The data generation unit 196, for example, transmits information relating to the generated embedding image data to the transmission / reception unit 192. The transmission / reception unit 192, for example, transmits information relating to the embedding image data received from the data generation unit 196 to the server 20. The reception control module 2031, for example, stores information relating to the embedding image data received from the terminal device 10 in the image data table 2021.

[0073] In step S16, the terminal device 10 accepts the designation of an object displayed in the embedded image (designation acceptance step). Specifically, for example, the operation acceptance unit 191 accepts a user instruction to instruct the output device 14 to present the embedded image. The operation acceptance unit 191, for example, transmits second acceptance information indicating that the above-mentioned user instruction has been accepted to the presentation control unit 193. For example, upon receiving the second acceptance information from the operation acceptance unit 191, the presentation control unit 193 generates instruction information instructing the reading of pixel data from the image data table 2021, i.e., the reading of the embedded image, and transmits the generated instruction information to the transmission / reception unit 192. The transmission / reception unit 192 transmits the instruction information received from the presentation control unit 193 to the server 20, for example.

[0074] The reception control module 2031 transmits, for example, instruction information received from the terminal device 10 to the transmission control module 2032. For example, when the transmission control module 2032 receives instruction information from the transmission control module 2032, the transmission control module 2032 reads out pixel data (embedded image) and reference information (path) of the point cloud data 183 from the image data table 2021 and transmits them to the terminal device 10.

[0075] The transmitting / receiving unit 192, for example, transmits the embedded image received from the server 20 to the presentation control unit 193, and temporarily stores the reference information (path) of the point cloud data 183 received from the server 20 in the storage unit 180. The presentation control unit 193, for example, transmits the embedded image received from the transmitting / receiving unit 192 to the output device 14, and displays it on the display screen of the display 141 as shown in Fig. 6. The operation accepting unit 191 accepts a user operation to specify the target image displayed on the display screen of the display 141, thereby accepting the user's specification of the target object.

[0076] Fig. 6 is a schematic diagram showing an example of the display screen of display 141 when an embedded image is presented to the user. In the example of Fig. 6, image 1411 is displayed as an embedded image on the display screen of display 141. Also, in the example of Fig. 6, image 1412 representing the length direction and width direction of the object is included in image 1411 as an object image.

[0077] For example, the user taps image 1412 displayed on the display screen of display 141 to specify the object displayed in image 1411. The operation accepting unit 191 accepts the user's specification of the object by accepting, for example, the user's tap operation on image 1412. The operation accepting unit 191 transmits, for example, third acceptance information indicating that the specification of the object has been accepted to the presentation control unit 193.

[0078] 5, the terminal device 10 executes the process of step S15 and the process of step S16 consecutively, but this is not limiting. For example, the terminal device 10 may execute the processes up to step S15 and the processes from step S16 onwards separately.

[0079] In step S17, the terminal device 10 reads point data of the specified object from the point cloud data 183 incorporated in the metadata of the image data 182 (reading step). Specifically, for example, upon receiving the third reception information, the presentation control unit 193 acquires the point cloud data 183 from an external storage area based on reference information (path) of the point cloud data 183 temporarily stored in the storage unit 180. The presentation control unit 193 extracts point data associated with the target image from the acquired point cloud data 183, and thereby reads the point data from the point cloud data 183.

[0080] It is not essential that the reference information (path) of the point cloud data 183 is temporarily stored in the storage unit 180. For example, in the process of reading point data, the presentation control unit 193 may use the reference information (path) of the point cloud data 183 that is held in the image data table 2021.

[0081] In step S18, the terminal device 10 acquires the size of the specified object based on the read point data (size acquisition step). Specifically, for example, the presentation control unit 193 reads size information of the object from the size information included in the point cloud data 183 acquired from the external storage area by referring to the read point data. As a result, the presentation control unit 193 acquires each size of the object based on the size information.

[0082] In step S19, the terminal device 10 presents the size of the acquired object (presenting step). Specifically, for example, the presentation control unit 193 projects the read point data onto the target image in the embedded image, which is two-dimensional data. This projection process causes the two-dimensional point data to overlap with the target image. Here, the presentation control unit 193 can perform this projection process because the point data is associated with the target image in the image indicated by the image data 182 in step S14. Hereinafter, the point data that has been two-dimensionally converted and projected onto the target image will be referred to as "two-dimensional point data."

[0083] The presentation control unit 193, for example, specifies each dimension of the outer shape corresponding to the two-dimensional point data based on each size of the acquired object. The presentation control unit 193, for example, displays each dimension of the outer shape corresponding to the two-dimensional point data specified based on each size of the object on the display screen of the display 141.

[0084] There are no particular limitations on the display mode of each dimension. For example, as shown in Fig. 7, the presentation control unit 193 may superimpose each dimension on the target image (two-dimensional point data) in a manner similar to writing dimensions on a drawing. "Similar to writing dimensions on a drawing" refers to, for example, a mode in which dimension lines, dimension values, dimension extension lines, etc. are displayed near the corresponding part of the target image (two-dimensional point data).

[0085] Fig. 7 is a schematic diagram showing an example of the display screen of display 141 when the size of an object is presented to the user. In the example of Fig. 7, the dimensions of each corresponding portion of the object in the length direction and width direction are superimposed around image 1412. Also, in the example of Fig. 7, the thickness of the corresponding portion of the object, which cannot be represented by image 1412 (two-dimensional point data), which is two-dimensional data, is displayed in text form separately from the superimposed display of the dimensions.

[0086] 5, the processes from step S16 to step S19 are not essential among the processes executed by the terminal device 10. The terminal device 10 only needs to execute the processes from step S11 to step S15.

[0087] [5 Summary] As described above, in this embodiment, the operation receiving unit 191 receives a user instruction to instruct the camera 160 to capture an image of the target area. When the operation receiving unit 191 receives the user instruction, the camera control unit 194 causes the camera 160 to capture an image of the target area and acquires image data 182 output from the camera 160. When the operation receiving unit 191 receives the user instruction, the sensor control unit 195 causes the 3D sensor 165 to measure the target area and generate point cloud data 183, and acquires the point cloud data 183 output from the 3D sensor 165. The data generation unit 196 associates each pixel constituting the target image extracted from the image data 182 with each point constituting the point data extracted from the point cloud data 183. The data generation unit 196 adds reference information (path) read from the point cloud data 183 to the metadata of the image data 182 received from the camera control unit 194, thereby generating embedded image data. The data generating unit 196 stores the generated embedded image data in the image data table 2021 via the transmitting / receiving unit 192 and the reception control module 2031 .

[0088] This eliminates the need to compare information about the length of the object shown in the image represented by image data 182 with information about the length of the object represented by the point data in point cloud data 183 to determine whether they match when associating the target image with the point data. Therefore, by simply associating the target image with the point data, it is possible to easily associate image data with the point cloud data.

[0089] Furthermore, in this embodiment, the operation accepting unit 191 accepts a user operation to specify a target image in the embedded image presented by the output device 14, thereby accepting a user's designation of an object. When the operation accepting unit 191 accepts the above-mentioned user operation, the presentation control unit 193 reads point data from the point cloud data 183 acquired from an external storage area. The presentation control unit 193 acquires the size of the object by reading size information of the object from the size information included in the point cloud data 183 acquired from the external storage area. The presentation control unit 193 presents the acquired size of the object.

[0090] This allows the user to grasp the size of the target object from the display screen of the display 141 simply by specifying the target image in the embedded image presented by the output device 14. This improves user convenience when using embedded image data.

[0091] [6. Modifications] In the present embodiment, an example has been described in which the terminal device 10 accepts the designation of a target image in an embedded image and outputs the size of the target object to the output device 14. Here, the terminal device 10 may, for example, analyze the embedded image data to determine whether or not the target object displayed in the embedded image (specifically, the target image) is abnormal.

[0092] Specifically, for example, the control unit 190 may function as a determination unit (not shown). The determination unit may, for example, perform image analysis of the embedded image to determine whether the object is abnormal. There are no particular limitations on the determination criteria for determining whether the object is abnormal, and any determination criteria may be set as long as they can be understood from the embedded image data. In this modified example, the determination unit sets the size of the object as the determination criterion. Furthermore, the determination unit outputs two types of determination results for "whether the object is abnormal or not," namely, a determination result that the object is normal (normal) and a determination result that the object is abnormal. The determination unit may, for example, further output a determination result that the object is in good condition (good condition) depending on the content of the determination criterion. "Good condition (good condition)" means that a certain state / situation is determined to be acceptable beyond a certain standard considered to be "normal (usual)."

[0093] More specifically, for example, the determination unit may perform image analysis of the embedded image and identify the sizes of the objects displayed in the embedded image (specifically, the target image) when the operation reception unit 191 receives an instruction to perform an abnormality determination from the user. There are no particular limitations on the image analysis method, and for example, a known method may be adopted. Instead of performing image analysis of the embedded image, the determination unit may analyze size information of the point cloud data 183 linked to the embedded image data to identify the sizes of the objects.

[0094] The determination unit may, for example, determine whether an object is abnormal by comparing each identified size with a size distribution. The size distribution indicates the distribution of size information contained in each of the multiple point cloud data 183 generated in the past for a specific size of the object. In other words, a different size distribution exists corresponding to each size of the object. For example, the determination unit may determine that an object is abnormal if there are a certain number or more object sizes that significantly deviate from the size distribution.

[0095] Instead of the size distribution, the determination unit may set, for example, a reference value corresponding to each of the identified sizes. In this case, the determination unit may determine whether or not the object is abnormal by comparing each identified size with the reference value. The reference value may be set to any value depending on the type and size of the object, the purpose of the abnormality determination, the desired determination accuracy, etc. The size distribution or the reference value may be stored in advance in the storage unit 180, for example, or may be input by the user to the input device 13.

[0096] The presentation control unit 193 may, for example, receive the determination result from the determination unit and transmit it to the output device 14, thereby displaying the determination result on the display screen of the display 141. This allows the user to easily understand the determination result simply by visually checking the display screen of the display 141.

[0097] In this way, in this modification, it is possible to determine whether or not an object is abnormal using the embedded image data, which broadens the range of uses for the embedded image data and improves user convenience.

[0098] [7 Basic Computer Hardware Configuration] 8 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 901, a main memory device 902, an auxiliary memory device 903, and a communication IF 991 (interface), which are electrically connected to one another by a communication bus 921.

[0099] The processor 901 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

[0100] The main storage device 902 is a memory for temporarily storing programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0101] The auxiliary storage device 903 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0102] The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using wired or wireless communication standards.

[0103] The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks (e.g., Wi-Fi (registered trademark)) that can connect to the Internet via a predetermined access point. In the case of a wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), and Bluetooth (registered trademark). In the case of a wired connection, the network also includes a direct connection using a USB (Universal Serial Bus) cable, etc.

[0104] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration across multiple computers 90 and interconnecting them via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0105] [8 Basic Functional Configuration of Computer 90] The following describes the functional configuration of a computer realized by the basic hardware configuration (FIG. 8) of the computer 90. The computer includes at least the functional units of a control unit, a storage unit, and a communication unit.

[0106] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 interconnected via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0107] The control unit is realized by the processor 901 reading out various programs stored in the auxiliary storage device 903, expanding them in the main storage device 902, and executing processing in accordance with the programs. The control unit can realize functional units that perform various types of information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0108] The storage unit is realized by a main storage device 902 and an auxiliary storage device 903. The storage unit stores data, various programs, and various databases. Furthermore, the processor 901 can allocate a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 in accordance with the programs. Furthermore, the control unit can cause the processor 901 to execute processes for adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0109] A database refers to a relational database, which manages data sets called masters and tables in a tabular format structurally defined by rows and columns, by relating them to each other. In a database, a table is called a table, a master, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables and masters can be set and associated.

[0110] Typically, each table and each master has a column set as a primary key to uniquely identify a record, but setting a primary key to a column is not essential. The control unit can cause the processor 901 to add, delete, or update records in specific tables and masters stored in the storage unit according to various programs.

[0111] Furthermore, by storing data, various programs, and various databases in the storage unit, it can be considered that the information processing device and information processing system according to the present disclosure have been manufactured.

[0112] Note that the databases and masters in this disclosure may include any data structure in which information is structurally defined (such as a list, dictionary, associative array, or object). The data structure also includes data that can be considered as a data structure by combining data with functions, classes, methods, etc. written in any programming language.

[0113] The communication unit is realized by the communication IF 991. The communication unit realizes a function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 901 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0114] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

[0115] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, and Java (registered trademark).

[0116] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or the storage medium.

[0117] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in a memory.

[0118] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0119] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0120] Although several embodiments of the present disclosure have been described above, these embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are intended to be included in the scope of the inventions and their equivalents as defined in the claims, as well as in the scope and spirit of the inventions.

[0121] [9. Appendix] The matters described in the above embodiments will be supplemented below.

[0122] <Appendix 1> A program for operating a computer having a processor, a memory, an imaging unit that captures images using visible light, and a three-dimensional sensor that has the same orientation as the imaging unit, the program causing the processor to execute the following steps: receiving an instruction to capture an image; acquiring image data from the imaging unit based on the received imaging instruction; acquiring point cloud data from the three-dimensional sensor based on the received imaging instruction; associating a target image of an object contained in the image data with point data of the object contained in the point cloud data; and incorporating the point cloud data, in which the point data is associated with the target image, into metadata of the image data, and storing the image data with the point cloud data incorporated into the metadata.

[0123] <Appendix 2> A program described in Appendix 1 that causes a processor to execute the steps of accepting the designation of an object shown in an image, reading point data of the designated object from point cloud data incorporated in metadata, obtaining the size of the designated object based on the read point data, and presenting the obtained size.

[0124] <Appendix 3> A program described in (Appendix 1) or (Appendix 2) in which, in the matching step, the target image and point data are matched based on the coordinate values ​​and pixel values ​​contained in the image data and the coordinate values ​​and density information contained in the point cloud data.

[0125] <Appendix 4> In the matching step, the image data is analyzed to extract a target image of the object contained in the image data, and the extracted target image is matched with point data of the object grasped based on coordinate values ​​and density information contained in the point cloud data. (Appendix 1) or (Appendix 2) is a program described in the program.

[0126] <Appendix 5> A program described in any one of (Appendix 1) to (Appendix 4) that causes a processor to execute a step of analyzing image data having point cloud data incorporated into metadata and determining whether an object shown in the image is abnormal.

[0127] <Appendix 6> An information processing device comprising a control unit and a storage unit, wherein the control unit executes all steps in the program according to any one of (Supplementary Note 1) to (Supplementary Note 5).

[0128] <Appendix 7> A method executed by a computer having a processor and a memory, wherein the processor executes all steps in the program described in any one of (Appendix 1) to (Appendix 5).

[0129] <Appendix 8> A system comprising means for executing all steps in the program described in any one of (Appendix 1) to (Appendix 5). [Explanation of symbols]

[0130] 1. System 10...Terminal device 120…Communications Department 13...Input device 14...Output device 15...Memory 16…Storage 19...Processor 20...Server 22...Communication IF 23...Input / output interface 25…Memory 26…Storage 29...Processor

Claims

1. A program for operating a computer including a processor, a memory, an image capturing unit that captures an image using visible light, and a three-dimensional sensor having the same orientation direction as the image capturing unit, The program causes the processor to: receiving an instruction to capture the image; acquiring image data from the photographing unit based on the received photographing instruction; acquiring point cloud data from the three-dimensional sensor based on the received photographing instruction; a step of associating a target image of the target object included in the image data with point data of the target object included in the point cloud data; incorporating the point cloud data, in which the point data is associated with the target image, into metadata of the image data, and storing the image data in which the point cloud data has been incorporated into the metadata; A program that executes the following.

2. receiving a designation of the object shown in the image; reading the point data of the specified object from the point cloud data incorporated in the metadata; acquiring a size of the specified object based on the read point data; presenting the obtained size; The program according to claim 1 , which causes the processor to execute the following steps.

3. The program according to claim 1 , wherein in the matching step, the target image and the point data are matched based on coordinate values ​​and pixel values ​​contained in the image data and coordinate values ​​and density information contained in the point cloud data.

4. The program of claim 1, wherein in the matching step, the image data is analyzed to extract a target image of the object contained in the image data, and the extracted target image is matched with point data of the object grasped based on coordinate values ​​and density information contained in the point cloud data.

5. The program according to claim 1 , causing the processor to execute a step of analyzing the image data in which the point cloud data has been incorporated into the metadata, and determining whether the object shown in the image is abnormal.

6. 6. An information processing apparatus comprising a control unit and a storage unit, wherein the control unit executes all steps of the program according to claim 1.

7. A method executed by a computer having a processor and a memory, wherein the processor executes all the steps of the program of any one of claims 1 to 5.

8. A system comprising means for executing all steps in the program according to any one of claims 1 to 5.

Images