Detection device

Abstract

To provide an imaging device capable of reducing the processing load.SOLUTION: An imaging device includes: a main body part; a detection unit including an imaging unit that captures an image of a target object, and a ranging unit that detects distance from each point on a surface of the target object; an information calculation unit that is provided in the main body part and calculates at least one of shape information and texture information of the target object using detection results of the detection unit; and a communication unit that transmits calculation results of the information calculation unit.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a detection device, an information processing device, and a system.

Background Art

[0002] A technique has been proposed in which an object is imaged by a plurality of imaging devices, and a plurality of obtained images are input to a computer to acquire the three-dimensional shape of the object (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, when generating the three-dimensional shape of an object from a plurality of images obtained by imaging the object using an imaging device, it is necessary to transfer all the images from the imaging device to a computer and process those images by the computer. In this case, since all the captured images are transferred from the imaging device to the computer, the communication (e.g., communication processing and communication data volume, etc.) between the imaging device and the computer may increase. The present invention has been made in view of the above circumstances, and an object thereof is to provide an imaging device, an information processing device, and an imaging system capable of reducing the communication load.

Means for Solving the Problems

[0005] According to an aspect of the present invention, a detection device is provided comprising: a detection unit for detecting an object within a field of view; an information generation unit for generating shape information of the object based on the detection result of the detection unit; and a communication unit for receiving a request signal from an external device and transmitting the shape information to the external device, wherein the communication unit receives a request signal specifying a range narrower than the field of view, the information generation unit generates shape information for the range specified by the request signal, and the communication unit transmits the shape information for the range. According to an aspect of the present invention, a detection device is provided comprising: a detection unit for detecting an object within a field of view; an information generation unit for generating shape information of the object based on the detection result of the detection unit; and a communication unit for communicating with an external device, wherein the communication unit transmits information of characteristic points of the shape information to the external device, and after transmitting the information of characteristic points to the external device, receives a request signal specifying a range narrower than the field of view, the information generation unit generates shape information for the range specified by the request signal, and the communication unit transmits the shape information for the range. According to an aspect of the present invention, a detection device is provided comprising: a detection unit for detecting an object within a field of view; an information generation unit for generating shape information of the object based on the detection result of the detection unit; and a communication unit for communicating with an external device, wherein the communication unit transmits additional information to the external device used to integrate multiple sets of shape information generated by detecting an object from multiple different positions, and after transmitting the additional information to the external device, receives a request signal specifying a range narrower than the field of view, the information generation unit generates shape information for the range specified by the request signal, the communication unit transmits the shape information for the range, and the additional information is generated based on the shape information of the object. According to an aspect of the present invention, a detection device is provided comprising: a detection unit for detecting an object within a field of view; an information generation unit for generating shape information of the object based on the detection result of the detection unit; and a communication unit for communicating with an external device, wherein the communication unit transmits information of feature points of the shape information to the external device, transmits additional information to the external device used to integrate multiple sets of shape information generated by detecting an object from multiple different positions after transmitting the information of feature points to the external device, receives a request signal specifying a range narrower than the field of view determined based on information of feature points common to each set of shape information obtained based on the additional information, the information generation unit generates shape information for the range specified by the request signal, and the communication unit transmits the shape information for the range. According to an aspect of the present invention, a detection device is provided comprising: a detection unit that detects an object within a field of view from a first direction; an information generation unit that generates first shape information of the object based on the detection result of the detection unit; and a communication unit that communicates with an external device, wherein the communication unit transmits to an external device additional information that is generated based on a first position indicating the position of a feature point of the first shape and a second position indicating the position of a feature point of second shape information of the object generated based on the detection result detected from a second direction different from the first direction, and is used to integrate multiple shape information generated by detecting the object from multiple different positions, and after transmitting the additional information to the external device, receives a request signal specifying a range narrower than the field of view determined based on feature point information common to each shape information obtained based on the additional information, the information generation unit generates shape information of the range specified by the request signal, the communication unit transmits the shape information of the range, and the additional information is generated based on the shape information of the object.

[0006] According to an aspect of the present invention, an information processing device is provided, comprising: a first detection device that detects an image of an object and the distance from the object and generates first shape information indicating the shape of a part of the surface of the object, not the whole surface; a second detection device that detects an image of an object and the distance from the object and generates second shape information indicating the shape of a part of the surface of the object, not the whole surface; and a communicating information processing device comprising: a receiving unit that receives additional information indicating the positional relationship between the first shape information and the second shape information; and a transmitting unit that transmits a request signal to the first detection device requesting first shape information within a range determined based on the additional information from the first shape information, and a request signal to the second detection device requesting second shape information within a range determined based on the additional information from the second shape information.

[0007] According to an aspect of the present invention, a system is provided comprising: a first detection device that detects an object from a first position and generates first shape information of the object; a second detection device that can communicate with the first detection device and detects an object from a second position and generates second shape information of the object; and an information processing device that can communicate with the first and second detection devices and generates integrated shape information by integrating the first shape information and the second shape information, wherein the information processing device transmits to the first detection device information indicating the portion of the first shape information necessary for generating the integrated shape information, and transmits to the second detection device information indicating the portion of the second shape information necessary for generating the integrated shape information. According to an aspect of the present invention, a first detection device detects an object from a first position, calculates first model information of the detected object, detects a first feature point which is a part of the shape indicated by the calculated first model information that can be distinguished from other parts, and calculates first feature point data which shows the position of the detected first feature point and the feature quantity of the feature point; and a second detection device detects an object from a second position different from the first position, calculates second model information of the detected object, detects a second feature point which is a part of the shape indicated by the calculated second model information that can be distinguished from other parts, and shows the position of the detected second feature point and the feature quantity of the feature point. The system comprises a second detection device that calculates second feature point data, and an information processing device that generates integrated model information by integrating first model information and second model information. The first detection device, the second detection device, and the information processing device are able to communicate with each other. The first detection device acquires second feature point data from the second detection device and calculates first additional information, including the positional relationship between the shape shown by the first model information and the information shown by the second model information, from the first feature point data and the acquired second feature point data. The second detection device acquires first feature point data from the first detection device and calculates first additional information, including the positional relationship between the shape shown by the first model information and the information shown by the second model information. From the point data, the information processing device calculates second additional information including the positional relationship between the shape indicated by the first model information and the shape indicated by the second model information. The information processing device acquires the first additional information from the first detection device and the second additional information from the second detection device. Based on the acquired first and second additional information, it determines the areas to be handled by the first model information from the first detection device and the second areas to be handled by the second model information from the second detection device, so as not to overlap. It then sends a command to the first detection device requesting the first designated information, which is the first model information in the first area to be handled, and the second detection device... A system is provided in which a command is sent to the first detection device requesting second designated information, which is second model information in the second area of ​​responsibility; the first detection device extracts the first designated information from the first model information based on the command from the information processing device and sends the extracted first designated information to the information processing device; the second detection device extracts the second designated information from the second model information based on the designation from the information processing device and sends the extracted second designated information to the information processing device; and the information processing device integrates the first designated information received from the first detection device and the second designated information received from the second detection device to generate integrated model information. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an imaging device, an information processing device, and an imaging system that can reduce the communication load. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing the imaging system according to the first embodiment. [Figure 2] This is a block diagram showing the imaging system according to the first embodiment. [Figure 3] This is a diagram showing the detection unit according to the first embodiment. [Figure 4] It is a diagram showing the operation of the imaging system according to the first embodiment. [Figure 5] It is a block diagram showing the imaging system according to the second embodiment. [Figure 6] It is a diagram showing the operation of the imaging system according to the second embodiment. [Figure 7] It is a diagram showing the imaging system according to the third embodiment. [Figure 8] It is a block diagram showing the imaging system according to the third embodiment. [Figure 9] It is a diagram showing the operation of the imaging device according to the third embodiment. [Figure 10] It is a diagram showing the operation of the imaging system according to the fourth embodiment. [Figure 11] It is a block diagram showing the imaging system according to the fifth embodiment. [Figure 12] It is a diagram showing the operation of the imaging system according to the fifth embodiment. [Figure 13] It is a block diagram showing the imaging system according to the sixth embodiment. [Figure 14] It is a block diagram showing the imaging system according to the seventh embodiment. [Figure 15] It is a diagram showing the operation of the imaging system according to the seventh embodiment. [Figure 16] It is a diagram showing the operation of the imaging system according to the eighth embodiment. [Figure 17] It is a diagram showing the operation of the imaging system according to the ninth embodiment.

Embodiments for Carrying Out the Invention

[0010] [First Embodiment] The first embodiment will be described. FIG. 1 is a diagram showing the imaging system 1 according to the present embodiment. For example, the imaging system 1 includes an imaging device 2, an information processing device 3, an input device 4, and a display device 5.

[0011] The imaging device 2 images the object OB, which is illuminated by, for example, light from a lighting device, indoor lighting, or natural light, and acquires image data of the object OB (e.g., visible light image, infrared light image). The imaging device 2 also detects the distance from the imaging device 2 to each point on the surface of the object OB and acquires depth information of the object OB. The depth information includes, for example, information relating the position of a point on the surface of the object OB to the distance (depth) from this point to the imaging device 2. The depth information also includes, for example, information showing the distribution of depth on the object OB (e.g., depth map).

[0012] The imaging device 2 uses the image data captured by the imaging unit 15 and the depth information detected by the distance measuring unit (distance detection unit) 16 to perform calculation processing on information related to the object OB. Through calculation processing, the imaging device 2 models at least a portion of the object OB and calculates model information (model data). For example, the imaging device 2 performs computer graphics processing (CG processing) on ​​at least a portion of the object OB through calculation processing and calculates model information (e.g., CG model data). The model information includes, for example, shape information that shows the three-dimensional shape of the object OB and texture information that shows the surface pattern of the object OB. Also, for example, the model information includes at least one of three-dimensional point coordinates, related information of those point coordinates, texture information of the surface defined by the point coordinates and related information, spatial information of the image such as the lighting conditions and light source information of the entire image, and polygon data as shape information.

[0013] For example, the object OB in Figure 1 includes a prismatic member and a soccer ball, and the shape information includes information about the surface of the prismatic member (e.g., a plane) and information about the surface of the soccer ball (e.g., a sphere). The texture information includes, for example, at least one piece of information such as letters or shapes on the surface of the soccer ball (e.g., black pentagons, white hexagons), patterns, information defining the unevenness, a specific image, and color (e.g., chromatic, achromatic).

[0014] The imaging device 2 calculates model information for the portion of the object OB that falls within the field of view of the imaging device 2 (field of view area). This model information is, for example, information for a model representing a part of a model representing the entire circumference of the object OB (hereinafter referred to as the full-circumference model). The imaging device 2 can supply at least a portion of the information calculated by the arithmetic processing (e.g., model information) to an external device. For example, the imaging device 2 supplies at least a portion of the model information to the information processing device 3.

[0015] Furthermore, for example, the imaging device 2 can output at least a portion of the model information to a digital device capable of inputting and outputting digital information such as barcodes or 2D codes. Such a digital device can display or print digital information, including at least a portion of the model information, on a display or on paper. A reader device equipped with a reader unit (e.g., an optical reader) that can read the displayed or printed digital information can input the digital information into the device's memory area via the reader unit. The reader device may also further include a rendering processing unit as described later. Note that the imaging system 1 described above may be configured to include the above-mentioned digital device and reader device equipped with the reader unit. Similarly, the imaging device 2 may be configured to include the above-mentioned digital device and reader device.

[0016] The information processing device 3 includes, for example, a computer system. The input device 4 includes, for example, at least one of the following: a keyboard, mouse, touch panel, sensor such as an accelerometer, voice input device, touch pen, etc., and is connected to the information processing device 3. The input device 4 receives information input from, for example, a user and supplies the input information to the information processing device 3. The display device 5 includes, for example, a liquid crystal display or a touch panel display, and is connected to the information processing device 3. The display device 5 displays an image using, for example, image data supplied from the information processing device 3.

[0017] The information processing device 3 is connected to the imaging device 2 via wired or wireless means, enabling communication. For example, the information processing device 3 may be connected to the imaging device 2 via a communication cable, or via an internet connection. Alternatively, the information processing device 3 may be able to communicate with the imaging device 2 via radio waves or infrared light.

[0018] The information processing device 3 acquires information from the imaging device 2 through communication with the imaging device 2. The information processing device 3 uses the information acquired from the imaging device 2 (e.g., model information) to perform rendering processing. For example, the information processing device 3 calculates estimated image data of the object OB viewed from this viewpoint based on the viewpoint (imaging direction) setting information input by the user to the input device 4. The information processing device 3, for example, supplies the estimated image data to the display device 5 and displays the estimated image on the display device 5.

[0019] The imaging device 2 extracts information used for rendering by the information processing device 3 from, for example, the data and depth information of the captured image. The information extracted by the imaging device 2 is, for example, at least a part of the model information. Since the imaging device 2 supplies the extracted information to the information processing device 3, the communication load between the imaging device 2 and the information processing device 3, and the processing load on the information processing device 3 can be reduced compared to, for example, supplying the data and depth information of the captured image to the information processing device 3.

[0020] The imaging device 2 may be, for example, a portable information terminal, a fixed-point camera, or a camera whose field of view can be changed manually or automatically. The imaging system 1 may include, for example, a moving device for moving the imaging device 2. This moving device may change the field of view of the imaging device 2 by moving it. The control unit that controls this moving device may be provided in the imaging system 1, for example, in the imaging device 2, or in the information processing device 3. Alternatively, the control unit that controls this moving device may be provided in an external device to the imaging system 1.

[0021] Next, the parts of the imaging system 1 will be described. Figure 2 is a block diagram of the imaging system 1 according to this embodiment. The imaging device 2 comprises a detection unit 11, an information calculation unit 12, a communication unit 13, and a main unit 14. The main unit 14 is, for example, a camera body, case, or housing. The detection unit 11, the information calculation unit 12, and the communication unit 13 are provided in the main unit 14.

[0022] The detection unit 11 comprises an imaging unit 15 and a distance measuring unit 16. Figure 3(A) shows an example of the detection unit 11. The imaging unit 15 comprises an imaging optical system 17 and an image sensor 18. The image sensor 18 is, for example, a CMOS image sensor or a CCD image sensor in which a plurality of pixels are arranged two-dimensionally. The image sensor 18 is housed in the main body 14. The image sensor 18 is, for example, sensitive to the wavelength band of visible light (e.g., 380 nm to 750 nm) or the wavelength band of infrared light. The imaging result of the image sensor 18 includes, for example, information on the gradation value for each color of each pixel (e.g., RGB data). The image sensor 18 outputs the imaging result (detection result) in the data format of a full-color image. A full-color image is, for example, an image in which the red (R), green (G), and blue (B) of each pixel are represented by gradation values ​​(e.g., 256 gradations).

[0023] The imaging optical system 17 includes, for example, multiple lenses and forms an image of an object plane (e.g., object OB) on the image sensor 18. The imaging optical system 17 is, for example, held in a lens barrel and attached to the main body 14 together with the lens barrel. The imaging optical system 17 and the lens barrel are, for example, interchangeable lenses and are removable from the main body 14. The lens barrel may be part of the main body 14 or it may not be removable from the main body 14.

[0024] The distance measuring unit 16 detects the distance from each point on the surface of the object OB. The distance measuring unit 16 includes, for example, a Time of Flight (ToF) sensor and detects the distance using the ToF method. Alternatively, the distance measuring unit 16 includes, for example, a phase difference sensor and detects the distance using the phase difference method. The distance measuring unit 16 comprises an illumination unit 19, an imaging optical system 20, an image sensor 21, and a controller 22.

[0025] The irradiation unit 19 is capable of irradiating the target object OB with infrared light. The irradiation unit 19 is controlled by the controller 22. The controller 22 changes the intensity of the infrared light emitted from the irradiation unit 19 over time (e.g., by amplitude modulation).

[0026] The image sensor 21 is, for example, a CMOS image sensor or a CCD image sensor in which multiple pixels are arranged two-dimensionally. The image sensor 21 is housed in the main body 14. The imaging optical system 20 includes, for example, multiple lenses and forms an image of the object surface (e.g., object OB) on the image sensor 21. The imaging optical system 20 may be an interchangeable lens or may be built into the main body 14. The image sensor 21 is sensitive to at least the wavelength range of light emitted by the illumination unit 19. For example, the image sensor 21 is controlled by the controller 22 and detects infrared light reflected and scattered by the object OB.

[0027] The controller 22 uses the detection result detected by the image sensor 21 of the distance measuring unit 16 to detect the distance from the surface of the object OB to the image sensor 21. For example, if the depth differs between a first position and a second position on the surface of the object OB, the light reflected and scattered at the first position will have a different flight distance (optical path) from the object OB to the image sensor 21 compared to the light reflected and scattered at the second position. Therefore, the phase at which the light reflected and scattered at the first position of the object OB enters the image sensor 21 is different from the phase at which the light reflected and scattered at the second position enters the image sensor 21. Since the controller 22 modulates the intensity of the light emitted from the illumination unit 19 over time, the intensity of the light incident on the image sensor 21 changes depending on the phase. The controller 22 calculates the depth based on the time change of the output of the image sensor 21, for example. The controller 22 calculates depth for each sub-region (e.g., one pixel, multiple pixels) of the image captured by the image sensor 21 from the distance between the image sensor 21 and the surface of the object OB, and associates the calculated depth (or distance) with the position of the sub-region. In this way, the controller 22 calculates depth information that includes the depth of each of multiple sub-regions, for example.

[0028] The illumination unit 19 may emit light including visible light, and the image sensor 21 may detect light reflected and scattered by the object OB. For example, the distance measuring unit 16 may detect the object OB using the image sensor 21 in both the state of light illumination by the illumination unit 19 and the state of no light illumination, and calculate depth information using the difference between the detection results in the two states.

[0029] The distance measuring unit 16 may also include a projector camera. In this case, the distance measuring unit 16 illuminates the object OB from the illumination unit 19 with light having a predetermined intensity distribution (e.g., patterned light, structured light, texture), and images the object OB with the image sensor 21. The controller 22 uses the intensity distribution of light on the object OB captured in the image taken by the image sensor 21 to detect the distance between the object OB and the distance measuring unit 16. The distance measuring unit 16 may also include a laser scanner. In this case, the distance measuring unit 16 illuminates the object OB with laser light from the illumination unit 19, scans the object OB with the laser light, and detects the laser light reflected from the object OB to detect the distance between the object OB and the distance measuring unit 16.

[0030] Figure 3(B) shows another example of the detection unit 11. The distance measuring unit 16 of this detection unit 11 detects the distance between the object OB and the detection unit 11 using the detection results obtained by detecting the object OB from multiple fields of view. The imaging unit 15 includes an imaging optical system 17a and an image sensor 18a, and an imaging optical system 17b and an image sensor 18b. The imaging optical system 17a and image sensor 18a image the object OB in a first field of view. The imaging optical system 17b and image sensor 18b image the object OB in a second field of view. The second field of view differs from the first field of view in at least one of the position of the viewpoint and the direction of the line of sight. The position of the viewpoint is, for example, the position of the image sensor 18a and the position of the image sensor 18b, where the image of the object is formed. The direction of the line of sight is, for example, parallel to the direction of the optical axis of the optical member located closest to the object among the optical members through which light toward the viewpoint passes. For example, the direction of the line of sight in the first field of view is coaxial with the optical axis on the exit side of the imaging optical system 17a. Similarly, the direction of the line of sight in the second field of view is coaxial with the optical axis on the exit side of the imaging optical system 17b. The distance measuring unit 16 uses the image captured by image sensor 18a and the image captured by image sensor 18b as a disparity image to detect the distance between the object OB and the detection unit 11.

[0031] Alternatively, the imaging unit 15 may capture images from the first field of view and images from the second field of view at different times using a single image sensor, and the distance measuring unit 16 may use these images to detect the distance between the object OB and the detection unit 11.

[0032] The distance measuring unit 16 may detect distances expressed as dimensional values ​​such as meters, or it may detect relative distances expressed as dimensionless values ​​normalized by a reference value. The distance measuring unit 16 may also be in a unitized form, for example, and may be externally attached to the main unit 14. At least a part of the optical system of the distance measuring unit 16 may be shared with at least a part of the optical system of the imaging unit 15. For example, in Figure 3(A), the distance measuring unit 16 may branch the infrared light from the light that has passed through the imaging optical system 17 using a dichroic mirror or the like, and detect the branched infrared light with the image sensor 21. In a projector camera, the illumination unit 19 may project a pattern onto the object OB via the imaging optical system 17.

[0033] Returning to the explanation of Figure 2, the information calculation unit 12 includes, for example, a digital signal processor (DSP). The information calculation unit 12 uses the detection results of the detection unit 11 to calculate at least one of the shape information and texture information of the object OB.

[0034] The information calculation unit 12 calculates shape information, including the coordinates of multiple points on the surface of the object OB (hereinafter referred to as point cloud data) and surface information including connection information between the multiple points. Surface information can be, for example, polygon data, vector data, or draw data. Connection information includes, for example, information relating the points at both ends of a line corresponding to a ridge (e.g., edge) of the object OB, and information relating multiple lines corresponding to the contour of a face of the object OB.

[0035] First, the information calculation unit 12 calculates point cloud data using the detection results (e.g., depth information) from the distance measuring unit 16 (point cloud data processing). For example, the information calculation unit 12 calculates point cloud data by performing a perspective transformation from the distance image indicated by the depth information to a planar image. If the fields of view of the imaging unit 15 and the distance measuring unit 16 are different, the information calculation unit 12 may, for example, convert the detection results of the distance measuring unit 16 into the result of detecting the object OB from the field of view of the imaging unit 15 by performing a perspective transformation (projection transformation). The information calculation unit 12 may, for example, perform a perspective transformation using parameters that depend on the positional relationship between the field of view of the imaging unit 15 and the field of view of the distance measuring unit 16 (e.g., viewpoint position, line of sight direction).

[0036] The information calculation unit 12 then estimates the plane between a point selected from multiple points included in the point cloud data and its neighboring points, and converts the point cloud data into polygon data that has planar information between points (surface processing). The information calculation unit 12 converts the point cloud data into polygon data using, for example, an algorithm using the least squares method. This algorithm may be, for example, an algorithm published in a point cloud processing library.

[0037] Next, the information calculation unit 12 calculates texture information, for example, using an inverse rendering method. The texture information includes, for example, pattern information showing the surface pattern of the object OB, light source information for the light illuminating the object OB, and optical property information showing the optical properties (e.g., reflectance, scattering rate) of the surface of the object OB. The light source information includes, for example, at least one item of information from the position of the light source, the direction in which light is irradiated from the light source to the object, the wavelength of the light irradiated from the light source, and the type of light source.

[0038] The information calculation unit 12 calculates light source information using, for example, a model that assumes Lambertian reflection, a model that includes albedo estimation, etc. For example, the information calculation unit 12 estimates the component of each pixel value in the image captured by the imaging unit 15 that originates from light diffused by the object OB and the component that is specularly reflected by the object OB. The information calculation unit 12 also calculates the direction in which light enters the object OB from the light source using, for example, the estimated result of the specularly reflected component by the object OB and shape information. The information calculation unit 12 also estimates the reflection characteristics of the object OB using, for example, the calculated light source information and shape information, and calculates optical characteristic information including the estimated result of the reflection characteristics. Furthermore, the information calculation unit 12 calculates pattern information by removing the influence of illumination light from the visible light image data using, for example, the calculated light source information and optical characteristic information.

[0039] Here, the imaging device 2 according to this embodiment further comprises a display unit 25, an input unit 26, a storage unit 27, and a control unit 28. The display unit 25 (see Figure 1) is, for example, a liquid crystal display or a touch panel display provided on the main unit 14. The display unit 25 displays, for example, the detection results of the detection unit 11 (e.g., a visible light image from the imaging unit 15), the calculation results of the information calculation unit 12 (e.g., a depth map), and at least a portion of various setting information.

[0040] The input unit 26 may include, for example, operation buttons on the main unit 14, a touch panel on the display unit 25, or a voice input device that recognizes the voice of a user. The input unit 26 may detect user operations and accept information input from the user. The input unit 26 transmits the input information to the control unit 28.

[0041] The storage unit 27 is a non-volatile memory such as a USB memory or memory card, and stores various types of information. The storage unit 27 may include a storage device built into the imaging device 2, or it may include a port to which a storage device that can be released from the imaging device 2 can be connected. In this embodiment, the information calculation unit 12 of the imaging device 2 generates model information that includes at least one of shape information and texture information, with header information (e.g., identification information such as a number or code). The communication unit 13 then transmits this model information. The header information includes at least one of the following: identification information, the position of the imaging device 2 (position information), the imaging timing by the imaging unit 15, the imaging time by the imaging unit 15, optical characteristic information of the object OB, and imaging environment information (e.g., light source information, illumination conditions for the object OB, etc.). In this way, the information calculation unit 12 of the imaging device 2 can generate model information having header information based on a predetermined data format and have the communication unit 13 transmit this model information.

[0042] The control unit 28 controls each part of the imaging device 2 by means of a command (control signal) from, for example, a user or an external device (e.g., an information processing device 3). For example, the control unit 28 causes the detection unit 11 to perform the above-mentioned detection process. This detection process includes, for example, imaging processing by the imaging unit 15 and distance detection processing by the distance measuring unit 16. The control unit 28 causes, for example, at least a portion of the detection results of the detection unit 11 to be stored in the storage unit 27. The control unit 28 causes, for example, the information calculation unit 12 to calculate model information. The control unit 28 causes, for example, at least a portion of the model information calculated by the information calculation unit 12 to be stored in the storage unit 27.

[0043] The storage unit 27 stores model information item by item, for example. For example, the storage unit 27 stores information for each item, such as shape information, texture information, light source information, optical properties information of the object OB, and pattern information, in separate data tables. The control unit 28 displays an image on the display unit 25 that shows at least a portion of the information stored in the storage unit 27, for example. The control unit 28 controls the communication unit 13 to transmit and receive information via the communication unit 13.

[0044] The communication unit 13 includes, for example, at least one of the following: an I / O port such as a USB port, or a communicator that performs wireless communication using radio waves or infrared rays. The communication unit 13 is controlled by the control unit 28, reads information stored in the storage unit 27, and transmits the read information to an external device. For example, the communication unit 13 transmits at least a portion of the calculation results (e.g., model information) of the information calculation unit 12 to the information processing device 3. The communication unit 13 also receives information, for example, commands from an external device. The communication unit 13 can store the received information in the storage unit 27 or supply the received information to the control unit 28. If the imaging device 2 is equipped with the digital device described above, the communication unit 13 may transmit at least a portion of the model information to the digital device. The digital device may then generate digital information based on the received model information and output the digital information to a medium such as paper.

[0045] As shown in Figure 2, the information processing device 3 comprises a communication unit 30, a storage unit 31, a rendering processing unit 32, and a control unit 33. The communication unit 30 includes, for example, at least one of a USB port, a network card, or a communication device that performs wireless communication using radio waves or infrared rays. The communication unit 30 can communicate with the communication unit 13 of the imaging device 2.

[0046] The storage unit 31 includes, for example, a removable storage medium such as a USB memory stick, or a large-capacity storage device such as an external or internal hard disk. The storage unit 31 stores, for example, at least some of the data of the information received via the communication unit 30, an imaging control program for controlling the imaging device 2, and processing programs for executing each process of the information processing device 3.

[0047] The rendering processing unit 32 includes, for example, a graphics processing unit (GPU). Alternatively, the rendering processing unit 32 may be configured such that the CPU and memory execute each process according to an image processing program. The rendering processing unit 32 performs, for example, at least one of the following processes: drawing, texture mapping, and shading.

[0048] The rendering processing unit 32 can, for example, calculate an estimated image (e.g., a reconstructed image) of the shape defined in the shape information of the model information viewed from an arbitrary viewpoint during the drawing process. In the following description, the shape indicated by the shape information is referred to as the model shape. The rendering processing unit 32 can, for example, reconstruct the model shape (e.g., an estimated image) from the model information (e.g., shape information) through the drawing process. The rendering processing unit 32 stores the calculated estimated image data in the storage unit 31. Since the imaging device 2 can transmit at least a portion of the model information to the information processing device 3, for example, the information processing device 3 can reduce the rendering processing load. Also, for example, the imaging device 2 does not need to transmit all the images captured by the imaging unit 15 to the information processing device 3, but transmits at least a portion of the model information calculated by the information calculation unit 12 (e.g., shape information and texture information) to the information processing device 3. For this reason, the imaging device of this embodiment can reduce the communication load of the information required for the drawing process of the rendering processing unit 32.

[0049] Furthermore, in the texture mapping process, the rendering processing unit 32 can calculate an estimated image by applying, for example, the image indicated by the texture information of the model information to the surface of an object in the estimated image. The rendering processing unit 32 can also calculate an estimated image by applying a texture different from the object's out-of-bounds (OB) to the surface of an object in the estimated image.

[0050] The rendering processing unit 32 can calculate an estimated image in which, in shading processing, shadows formed by light sources indicated by the light source information of the model information are added to the objects in the estimated image. Furthermore, the rendering processing unit 32 can calculate an estimated image in which shadows formed by arbitrary light sources are added to the objects in the estimated image in shading processing.

[0051] The control unit 33 controls, for example, each part of the information processing device 3, the imaging device 2, the input device 4, and the display device 5. The control unit 33 controls, for example, the communication unit 30 to transmit commands (control signals) and setting information to the imaging device 2. The control unit 33 stores, for example, the information received by the communication unit 30 from the imaging device 2 in the storage unit 31. The control unit 33 controls, for example, the rendering processing unit 32 to execute the rendering process.

[0052] The control unit 33 controls the imaging device 2 by sending commands (signals) to the imaging device 2, for example, via the communication unit 30. For example, the control unit 33 controls the communication unit 30 and sends a command (request signal) to the imaging device 2 requesting the transmission of predetermined information. The control unit 33 may also send commands to the imaging device 2 to cause the imaging device 2 to perform various processes. For example, the control unit 33 may send a command to the detection unit 11 of the imaging device 2 to perform a detection process. For example, the control unit 33 may send a command to the information calculation unit 12 of the imaging device 2 to perform a model information calculation process. Furthermore, the control unit 33 of the information processing device 3 may send a command to the imaging unit 15 of the imaging device 2 to perform imaging of the object OB, and a command to set the imaging conditions for the object OB, to the imaging device 2.

[0053] The communication unit 13 selectively transmits information calculated by the information calculation unit 12, item by item. For example, the setting information stored in the storage unit 27 includes transmission item information that determines whether or not to transmit information for each item of the model information, and transmission order information that determines the order in which the information for each item is transmitted. This setting information can be updated, for example, by operation of the input unit 26 or by commands from the information processing device 3. The control unit 28 controls the communication unit 13, for example, to transmit the information for the items specified in the transmission item information in the order specified in the transmission order information. Alternatively, the control unit 28 may control the communication unit 13 to transmit the information for the items specified in the transmission item information (e.g., shape information and texture information) all at once based on a predetermined data format.

[0054] The transmitted item information may be set according to whether or not it is used in the rendering process of the information processing device 3. For example, in the rendering process, a different texture may be composited onto the shape of the object OB. In this case, the information processing device 3 can perform the rendering process using the shape information of the object OB but without using the texture information of the object OB. The transmitted item information may be set to specify, for example, that shape information will be transmitted but texture information will not. In this case, the information calculation unit 12 does not need to calculate the texture information.

[0055] Furthermore, for example, in the rendering process, it may be necessary to calculate an image with modified lighting for the object's outbound path (OB). The information processing device 3 can perform the rendering process using, for example, the shape information, pattern information, and optical property information of the object's OB, but without using the light source information. In this case, the transmission item information is set to specify, for example, that the shape information, pattern information, and optical property information will be transmitted, but the light source information will not be transmitted. Alternatively, for example, the transmission item information is set to specify that at least one of the following information will be transmitted: shape information, texture information, pattern information, light source information, and optical property information. The imaging device 2 can reduce the communication load by transmitting, for example, only a portion of the model information.

[0056] The transmission order information may be set according to the priority in the rendering process of the information processing device 3, for example. For example, the transmission order information may be set to send information of items used earlier in the rendering process first. For example, in the rendering process, an image of an object OB without texture may be calculated while changing the viewpoint, and after determining the viewpoint, an image of the object OB with texture may be calculated from this viewpoint. For example, the information processing device 3 can calculate an image of an object OB without texture viewed from a different viewpoint, using shape information but not texture information. The transmission item information may be set to information that specifies that shape information is sent first and texture information is sent after shape information. For example, when the imaging device 2 transmits information of each item of model information in an order according to the priority in the rendering process of the information processing device 3, it can transmit that information in parallel with part of the rendering process of the information processing device 3. Also, for example, the control unit 28 of the imaging device 2 can execute at least part of the calculation of model information by the information calculation unit 12 and the information transmission process by the communication unit 13 in parallel.

[0057] The control unit 33 of the information processing device 3 stores, for example, the information input to the input device 4 in the storage unit 31. The information input to the input device 4 includes, for example, setting information for the rendering process. This setting information includes, for example, at least one of the following: data to be used for the drawing process (e.g., shape information of the model information), viewpoint information in the drawing process, data of the object to which the texture will be applied in the texture mapping process, information of the texture to be applied in the texture mapping process (e.g., texture information of the model information), and light source information in the shading process (e.g., light source information of the model information). The rendering processing unit 32 executes the rendering process according to the setting information, for example.

[0058] The control unit 33, for example, causes the display device 5 to display images representing various information stored in the memory unit 31. For example, the control unit 33 displays rendering process setting information on the display device 5 and accepts changes to the setting information via the input device 4. The control unit 33 also causes the display device 5 to display images representing estimated image data stored in the memory unit 31.

[0059] The information processing device 3 does not necessarily have to display the estimated image obtained by the rendering process on the display device 5, and in this case, the imaging system 1 does not have to be equipped with a display device 5. For example, the information processing device 3 may transmit at least a portion of the estimated image data calculated by the rendering process to another device (playback device) via the communication unit 30, and the other device may display this image. For example, the information processing device 3 may transmit the estimated image data to the communication unit 13 of the imaging device 2 via the communication unit 30, and the imaging device 2 may display the estimated image on the display unit 25 based on the estimated image data received via the communication unit 13. For example, the playback device acquires the information (estimated image) calculated by the rendering process and displays that information on the display unit.

[0060] The information processing device 3 may also receive various setting information from other devices via the communication unit 30, in which case the imaging system 1 does not need to have an input device 4. For example, the imaging device 2 may transmit rendering processing setting information (e.g., viewpoint information that will form the basis of the estimated image) to the communication unit 30 of the information processing device 3 via the communication unit 13. The information processing device 3 may also execute rendering processing according to the rendering processing setting information received from the imaging device 2.

[0061] The imaging device 2 may also send a command to the communication unit 30 of the information processing device 3 via the communication unit 13 requesting estimated image data calculated by rendering. The information processing device 3 may send the estimated image data to the communication unit 13 of the imaging device 2 via the communication unit 30 in response to the command from the imaging device 2. The imaging device 2 may send the above request command as part of the rendering process setting information, or it may send it as a separate command (e.g., a control signal) from the rendering process setting information.

[0062] Furthermore, when the imaging device 2 is controlled by the information processing device 3 to perform the above-mentioned various processes, it does not need to be equipped with at least some of the control unit 28, storage unit 27, display unit 25, and input unit 26. Also, the imaging device 2 may perform the above-mentioned various processes by user operation, for example, and may not be controlled by the information processing device 3. For example, the imaging device 2 may perform the above-mentioned various processes without receiving commands (control signals) from an external device, or it may perform the above-mentioned various processes by user operation or according to a predetermined processing schedule. The imaging device 2 may transmit the calculation results of the information calculation unit 12 (e.g., model information, etc.) to other devices of the information processing device 3 via the communication unit 13.

[0063] Next, the image processing method will be described based on the operation of the imaging system 1. Figure 4 is a flowchart of the operation of the imaging system 1. In step S1 of Figure 4, the imaging device 2 uses the imaging unit 15 to image the object OB and acquires image data (e.g., a visible light image) (imaging process). In step S2, the imaging device 2 acquires point cloud data showing the three-dimensional shape of the surface of the object OB (point cloud data processing). In step S2, the imaging device 2 uses the distance measuring unit 16 to detect the distance between the object OB and the imaging device 2 and acquires depth information (step S3). The information calculation unit 12 of the imaging device 2 calculates point cloud data from the depth information, for example, by perspective transformation (step S4). In step S4, for example, the information calculation unit 12 adjusts the parallax between the visible light image data and the depth information as needed (viewpoint transformation process). The information calculation unit 12 may perform the processing in step S2 in parallel with at least a part of the processing in step S1, or it may perform it before the start or after the completion of the processing in step S1.

[0064] In step S5, the information calculation unit 12 calculates (generates) model information (model information generation process). The information calculation unit 12 calculates surface information as shape information, for example (step S6) (surface processing). The information calculation unit 12 also calculates texture information using the calculated surface information by an inverse rendering method, for example (step S7). In step S7, the information calculation unit 12 calculates light source information (step S8). The information calculation unit 12 also calculates pattern information using the calculated light source information, for example (step S9). In step S5, the information calculation unit 12 stores the calculated information in the storage unit 27 for each item.

[0065] In step S10, the imaging device 2 transmits at least a portion of the model information stored in the storage unit 27 to the information processing device 3 via the communication unit 13. The information processing device 3, for example, uses the model information received from the imaging device 2 to perform rendering processing and displays the result (e.g., estimated image) on the display device 5.

[0066] [Second Embodiment] Next, a second embodiment will be described. In this embodiment, components similar to those in the above-described embodiment are denoted by the same reference numerals, and their descriptions are simplified or omitted. Figure 5 is a diagram showing the imaging system 1 according to this embodiment. The imaging device 2 according to this embodiment further comprises a model integration unit 35 and a position detection unit 36.

[0067] The model integration unit 35 of the imaging device 2 integrates first model information calculated based on the result of detecting the object OB from a first direction (first detection result) and second model information calculated based on the result of detecting the object OB from a second direction (second detection result) to generate integrated model information. In this embodiment, the information calculation unit 12 calculates the first model information and the second model information respectively using the techniques described above.

[0068] The model integration unit 35 integrates the first model information and the second model information by, for example, associating the first feature point of the shape shown by the first model information with the second feature point of the shape shown by the second model information. The first feature point is a part of the shape shown by the first model information that can be distinguished from other parts. For example, a part defined as a surface in surface information can be distinguished from other surfaces by the shape of its outer perimeter, etc. The second feature point is a part of the shape shown by the second model information that can be distinguished from other parts.

[0069] For example, the model integration unit 35 detects a first feature point using at least one of the shape information and texture information included in the first model information, and calculates first feature point data indicating the location of the first feature point and the feature quantity (first feature quantity) of that feature point. The model integration unit 35 stores the first feature point data in the storage unit 27 as one item of information in the first model information. The model integration unit 35 also detects a second feature point using at least one of the shape information and texture information included in the second model information, and calculates second feature point data indicating the location of the second feature point and the feature quantity (second feature quantity) of that feature point. The model integration unit 35 stores the second feature point data in the storage unit 27 as one item of information in the second model information. The model integration unit 35 calculates feature points and feature quantities using, for example, the SIFT (Scale-Invariant Feature Transform) method or the SURF (Speeded Up Robust Features) method. For example, the feature point data mentioned above (e.g., first feature point data, second feature point data) is data that includes at least the feature quantity of the feature point, and may also be data that includes only the feature quantity of the feature point and the feature quantity.

[0070] The model integration unit 35 calculates integrated model information that shows third feature point data obtained by matching and integrating first feature point data having the first feature quantity of the first model information and second feature point data having the second feature quantity of the second model information. The third feature point data is, for example, data obtained by matching first feature point data (e.g., first feature quantity) and second feature point data (e.g., second feature quantity). The third feature point data is, for example, integrated data that includes the part of the object OB where the first feature point data and the second feature point data overlap, and models a wider range than either the first partial shape or the second partial shape.

[0071] For example, if the imaging device 2 detects the object OB from one side in an arbitrary first direction and calculates first model information, and then detects the object OB from the opposite side in that first direction and calculates second model information, the model integration unit 35 can calculate integrated model information that covers almost the entire circumference of the object OB. Note that the shape shown by the integrated model information may be a partial model showing only a part of the object OB.

[0072] Here, the integrated model information includes, for example, at least one of shape information and texture information. The shape information of the integrated model information is, for example, information obtained by integrating the shape information of the first model information and the shape information of the second model information. The texture information of the integrated model information is, for example, information obtained by integrating the texture information of the first model information and the texture information of the second model information.

[0073] Furthermore, the position detection unit 36 ​​detects the position information of the imaging device 2 (e.g., detection unit 11) based on a command (detection request signal) from the control unit 28, for example. The position detection unit 36 ​​includes, for example, at least one of a compass sensor, a GPS sensor, and an acceleration sensor. The position detection unit 36 ​​is provided, for example, inside the main body 14, but may also be attached externally to the main body 14. The position detection unit 36 ​​may also be a unit that is removable from the main body 14.

[0074] The position information of the detection unit 11 includes, for example, at least one of the detection direction of the detection unit 11 and global positioning information. The detection direction of the detection unit 11 is, for example, the direction of the optical axis of the imaging optical system 17 of the imaging unit 15 shown in Figure 3(A). The detection direction of the detection unit 11 is predetermined with respect to the attitude of the imaging device 2, and the position detection unit 36 ​​detects the detection direction of the detection unit 11 by detecting the attitude of the imaging device 2 using an orientation sensor. The detection unit 11 also detects global positioning information using, for example, a GPS sensor. The position information of the detection unit 11 also includes the amount of change in the position of the detection unit 11. The position detection unit 36 ​​can detect the velocity of the detection unit 11 by, for example, integrating the detection result of the acceleration sensor over time. The position detection unit 36 ​​can also detect the amount of change in the position of the detection unit 11 by, for example, integrating the velocity of the detection unit 11 over time. The detection unit 11 may, for example, detect position information with a first accuracy using a GPS sensor and obtain position information with a second accuracy that is more accurate than the first accuracy using the detection result of the acceleration sensor.

[0075] The control unit 28, for example, causes the detection unit 11 to detect an object OB, and the position detection unit 36 ​​to detect the position information of the detection unit 11. The control unit 28 stores the detection result of the detection unit 11 in the storage unit 27, associating it with the detection result of the position detection unit 36 ​​(the position information of the detection unit 11). The information calculation unit 12 calculates model information based on the detection result of the object OB by the detection unit 11, and stores the information of each item of this model information in the storage unit 27, associating it with the position information of the detection unit 11. The model information may include, for example, the position information of the detection unit 11.

[0076] The position information of the detection unit 11 is used, for example, to integrate the first model information and the second model information. For example, the model integration unit 35 may use the first position information of the detection unit 11 when the detection unit 11 detects the raw data of the first model information and the second position information of the detection unit 11 when the detection unit 11 detects the raw data of the second model information to match and integrate the first model information and the second model information. For example, the model integration unit 35 may use the first position information and the second position information to calculate the positional relationship between the first partial shape indicated by the first model information and the second partial shape indicated by the second model information. When the above position information is used to integrate the model information, the imaging device 2 or information processing device 3 that performs the integration can reduce the amount of computation required for integration and can also reduce mismatches in the integration.

[0077] The model integration unit 35 may integrate the first model information and the second model information using the location information of the detection unit 11 and feature point data including feature quantities. Alternatively, the model integration unit 35 may integrate the first model information and the second model information using the location information of the detection unit 11 but without using feature point data. For example, the model integration unit 35 may integrate the first model information and the second model information using feature point data (e.g., feature quantities) but without using the location information of the detection unit 11.

[0078] By the way, the region adjacent to the edge of the shape indicated by the shape information of the first model information (hereinafter referred to as the out-of-field region) includes, for example, a region that cannot be detected from the first direction. The out-of-field region is, for example, an unclosed part of the shape indicated by the first model information. The information calculation unit 12 or the model integration unit 35 may calculate the detection direction of the detection unit 11 that can acquire shape information of the out-of-field region. The imaging device 2 may display this detection direction on the display unit 25 as, for example, a candidate for the second direction after the detection process of the object OB from the first direction is completed.

[0079] Next, the image processing method will be described based on the operation of the imaging system 1. Figure 6 is a flowchart showing the operation of the imaging system 1. The imaging device 2 performs the process of step S11, which acquires at least the data of the captured image, with the detection direction of the detection unit 11 set to the first direction. In step S11, the imaging device 2 detects the object OB from the first direction using the detection unit 11 and acquires, for example, visible light image data and point cloud data. In step S12, the information calculation unit 12 calculates first model information using the detection result of the detection unit 11 in step S11. In step S13, the information calculation unit 12 stores the first model information in the storage unit 27.

[0080] In step S11, after the detection process by the detection unit 11 is completed, the imaging device 2 is moved, for example, by the user manually or by a moving device (not shown), and the detection direction of the detection unit 11 is set to a second direction different from the first direction. In step S14, the imaging device 2 detects the object OB from the second direction using the detection unit 11 and acquires, for example, visible light image data and point cloud data. In step S15, the information calculation unit 12 calculates second model information using the detection result of the detection unit 11 in step S14. In step S16, the information calculation unit 12 stores the second model information in the storage unit 27.

[0081] In step S17, the model integration unit 35 integrates the first model information and the second model information to calculate integrated model information. In step S17, the model integration unit 35 reads at least one of the shape information and texture information of the first model information from the storage unit 27 and detects a first feature point using at least one of the shape information and texture information (step S18). The model integration unit 35 then calculates first feature point data that includes at least the feature quantities of the first feature point (step S19) and stores the first feature point data in the storage unit 27 (step S20). The model integration unit 35 reads at least one of the shape information and texture information of the second model information from the storage unit 27 and detects a second feature point using at least one of the shape information and texture information (step S21). The model integration unit 35 then calculates second feature point data that includes at least the feature quantities of the second feature point (step S22) and stores the second feature point data in the storage unit 27 (step S23). The model integration unit 35 reads the first feature point data and the second feature point data from the storage unit 27 and matches these feature point data (step S24). The model integration unit 35 calculates integrated model information through the matching process in step S24 and stores the integrated model information in the storage unit 27 for each item (step S25). Then, in step S26, the imaging device 2 transmits at least a portion of the integrated model information stored in the storage unit 27 to the information processing device 3 via the communication unit 13. The information processing device 3, for example, performs rendering processing using the integrated model information received from the imaging device 2 and displays the result (e.g., estimated image) on the display device 5.

[0082] [Third Embodiment] Next, a third embodiment will be described. In this embodiment, components similar to those in the above-described embodiments are denoted by the same reference numerals, and their descriptions are simplified or omitted. Figure 7 is a diagram showing the imaging system 1 according to this embodiment. This imaging system 1 comprises a plurality of imaging devices 2. In the following description, one of the plurality of imaging devices 2 will be referred to as the first imaging device 2a, and the other imaging device 2 will be referred to as the second imaging device 2b. Therefore, this imaging system 1 is configured to include at least two imaging devices, or a plurality of imaging devices.

[0083] The first imaging device 2a detects the object OB from a first direction and calculates first model information based on the detection result. The first imaging device 2a supplies the first model information to the information processing device 3. The second imaging device 2b detects the object from a second direction and calculates second model information based on the detection result. The second imaging device 2b supplies the second model information to the information processing device 3. The information processing device 3 integrates the first model information and the second model information and calculates integrated model information based on detection results from multiple directions (e.g., a first direction and a second direction different from the first direction).

[0084] Next, the parts of the imaging system 1 will be described. Figure 8 is a block diagram of the imaging system 1. The first imaging device 2a includes a timing unit 41 (hereinafter referred to as timing unit 41a) that detects timing information (hereinafter referred to as timing information) for when the detection unit 11 (hereinafter referred to as detection unit 11a) detects an object OB. The timing information includes, for example, information indicating the period (e.g., date and time) during which the detection unit 11a performed the detection process. The timing information may include at least one of the start time and end time of the period during which the detection unit 11a performed the detection process.

[0085] Timing information is used, for example, when performing model integration using multiple detection results where the timing of object OB detection by the imaging device differs. For example, multiple model information may be calculated based on at least two detection results where the timing of object OB detection is relatively close, and the calculated multiple model information may be integrated.

[0086] The detection unit 11a stores the detection result in the storage unit 27 (hereinafter referred to as storage unit 27a) in association with timing information. The information calculation unit 12 (hereinafter referred to as information calculation unit 12a) calculates first model information based on the detection result of the detection unit 11a, and stores the calculation result in the storage unit 27a in association with timing information.

[0087] Furthermore, the second imaging device 2b has the same configuration as the first imaging device 2a, for example. The detection unit 11 (hereinafter referred to as the detection unit 11b) of the second imaging device 2b detects an object OB from a second direction, for example, and stores the detection result in the storage unit 27 (hereinafter referred to as the storage unit 27b) in association with the timing information detected by the timing unit 41 (hereinafter referred to as the timing unit 41b). The information calculation unit 12 (hereinafter referred to as the information calculation unit 12b) of the second imaging device 2b calculates second model information based on the detection result of the detection unit 11b, for example, and stores the calculation result in the storage unit 27b in association with the timing information.

[0088] The timing information may also include information about the period during which the detection unit 11 is scheduled to detect the target object OB. The first imaging device 2a and the second imaging device 2b may transmit and receive timing information and synchronize with each other using the control unit 28 and the communication unit 13 to detect the target object OB.

[0089] In this embodiment, the information calculation unit 12a of the first imaging device 2a calculates first additional information used for integrating the first model information and the second model information. The first additional information includes, for example, at least one of the following: information obtained by matching feature point data (e.g., feature quantities), position information of the detection unit 11a, timing information of the imaging device 2, position information of the detection unit 11b, and timing information of the second imaging device 2b. For example, the first imaging device 2a receives second model information from the second imaging device 2b, and the information calculation unit 12a of the first imaging device 2a calculates the first additional information using the calculated first feature point data and the second feature point data received from the second imaging device 2b.

[0090] In this embodiment, the information processing device 3 further includes a model integration unit 45. The model integration unit 45 integrates the first model information from the first imaging device 2a and the second model information from the second imaging device 2b to generate the integrated model information. The algorithm used by the model integration unit 45 for integrating the model information may be the same as, for example, the model integration unit 35 of the imaging device 2 described in the second embodiment. The communication unit 30 of the information processing device 3 receives, for example, first additional information from the communication unit 13a of the first imaging device 2a before performing model integration. The model integration unit 45 of the information processing device 2 performs model integration using, for example, the first model information, the second model information, and the first additional information.

[0091] Figure 9 shows the operation of the imaging system 1 according to this embodiment. The information calculation unit 12a of the first imaging device 2a calculates first model information in step S31 and calculates first feature point data including at least a first feature quantity in step S32. The information calculation unit 12b of the second imaging device 2b calculates second model information in step S33 and calculates second feature point data including at least a second feature quantity in step S34. The communication unit 13a of the first imaging device 2a sends a command (request command) to the communication unit 13b of the second imaging device 2b in step S35, requesting the transmission of the second feature point data. The communication unit 13b of the second imaging device 2b receives the request command from the communication unit 13a of the first imaging device 2a in step S36. The communication unit 13b transmits the second feature point data to the communication unit 13a in step S37. The communication unit 13a receives the second feature point data from the communication unit 13b in step S38. Thus, when communicating (data communication) at least feature point data (e.g., first feature point data and / or second feature point data) between imaging devices (e.g., between the first imaging device and the second imaging device), the information processing device 3 can reduce the integration load by having each imaging device calculate the information necessary for integrating the model information. It should be noted that the information processing device 3 can also create the above-mentioned feature point data (first feature point data, second feature point data, etc.). In this case, for example, the information processing device 3 generates the first and second feature point data using the model information received from each imaging device or one of the imaging devices (e.g., the first imaging device 2a). Alternatively, the first imaging device 2a (or the second imaging device 2b) can also generate the above-mentioned feature point data (e.g., first feature point data, second feature point data). In this case, for example, the first imaging device 2a generates the second feature point data based on the second model information received from the second imaging device 2b.

[0092] In step S39, the information calculation unit 12a of the first imaging device 2a matches the first feature point data and the second feature point data (feature portion matching process), and in step S40, calculates first additional information indicating the result of the matching. The first additional information includes, for example, information indicating the positional relationship between the shape indicated by the first model information and the shape indicated by the second model information. The first additional information includes, for example, the coordinates of feature points common to the first feature data and the second feature data, and / or feature quantities common to the first feature data and the second feature data. The information calculation unit 12a stores the calculated first additional information in, for example, the storage unit 27a. The first additional information may be, for example, information for one item of the first model information.

[0093] In step S41, the communication unit 13a of the first imaging device 2a transmits the first additional information to the communication unit 30 of the information processing device 3. In step S42, the communication unit 30 of the information processing device 3 receives the first additional information from the communication unit 13a. In step S43, the information processing device 3 determines and sets a first area to be handled by the first model information from the first imaging device 2a and a second area to be handled by the second model information from the second imaging device 2b, from the integrated model information. For example, the model integration unit 45 selects each area so that the first and second areas do not overlap.

[0094] In step S44, the control unit 33 of the information processing device 3 controls the communication unit 30 and sends a command (first request signal) to the communication unit 13a of the first imaging device 2a requesting the transmission of first model information in the first assigned area. Hereinafter, the first model information in the first assigned area will be referred to as first designated information. The first designated information is, for example, information about a part of the shape indicated by the first model information. In step S45, the communication unit 13a of the first imaging device 2a receives the command from step S44. In step S46, the information calculation unit 12a of the first imaging device 2a extracts the first designated information from the first model information. In step S47, the communication unit 13a of the first imaging device 2a transmits the first designated information to the communication unit 30 of the information processing device 3. In step S48, the communication unit 30 receives the first designated information from the communication unit 13a.

[0095] In step S49, following the processing in step S44, the control unit 33 of the information processing device 3 controls the communication unit 30 and sends a command (second request signal) to the communication unit 13b of the second imaging device 2b requesting the transmission of second model information in the second area of ​​responsibility. Hereinafter, the second model information in the second area of ​​responsibility will be referred to as second designated information. The second designated information is, for example, information about a part of the shape indicated by the second model information. In step S50, the communication unit 13b of the second imaging device 2b receives the command in step S49. In step S51, the information calculation unit 12b of the second imaging device 2b extracts the second designated information from the second model information. In step S52, the communication unit 13b of the second imaging device 2b transmits the second designated information to the communication unit 30 of the information processing device 3. In step S53, the communication unit 30 of the information processing device 3 receives the second designated information from the communication unit 13b. In step S54, the model integration unit 45 of the information processing device 3 integrates the first designated information, which is part of the first model information, and the second designated information, which is part of the second model information, to calculate integrated model information. For example, the model integration unit 45 generates integrated model information using information (e.g., additional information, designated information) obtained through the sharing of model information (e.g., first model information, second model information) or feature point data (e.g., first feature point data, second feature point data) between multiple imaging devices (e.g., information sharing by bidirectional communication, one-way communication). Also, for example, imaging device 2a and imaging device 2b perform data communication of information (e.g., first model information, second model information, first feature point data, second feature point data) between multiple imaging devices in order to create the above-mentioned additional information.

[0096] [Fourth Embodiment] Next, a fourth embodiment will be described. In this embodiment, components similar to those in the embodiments described above are denoted by the same reference numerals, and their descriptions are simplified or omitted. The imaging system 1 according to this embodiment has the same configuration as in Figures 7 and 8. In this embodiment, the first imaging device 2a and the second imaging device 2b each calculate additional information used for model integration. In this embodiment, the information processing device 3 checks the integrated model information and performs interpolation processing as appropriate.

[0097] Figure 10 is a diagram showing the operation of the imaging system 1 according to this embodiment. In Figure 10, the same reference numerals are used for processes similar to those in Figure 9, and their explanations are simplified or omitted. The information calculation unit 12a of the first imaging device 2a calculates first model information in step S31 and calculates first feature point data including at least a first feature quantity in step S32. The information calculation unit 12b of the second imaging device 2b calculates second model information in step S33 and calculates second feature point data including at least a second feature quantity in step S34. The communication unit 13a of the imaging device 2 transmits the first feature point data to the communication unit 13b of the second imaging device 2b in step S60. The communication unit 13b receives the first feature point data from the communication unit 13a in step S61. The communication unit 13b transmits the second feature point data to the communication unit 13a in step S62. In step S63, the communication unit 13a receives the second feature point data from the communication unit 13b.

[0098] In step S64, the information calculation unit 12a of the first imaging device 2a matches the first feature point data with the second feature point data and calculates first additional information indicating the matching result. In step S65, the communication unit 13a of the first imaging device 2a transmits the first additional information to the communication unit 30 of the information processing device 3. In step S66, the communication unit 30 receives the first additional information from the communication unit 13a.

[0099] The information calculation unit 12b of the second imaging device 2b matches the first feature point data and the second feature point data, similar to the first imaging device 2a, and calculates second additional information indicating the matching result in step S67. The information calculation unit 12b stores the calculated second additional information in the storage unit 27b. The second additional information is the same as the first additional information, and may, for example, be information for one item of the second model information. The communication unit 13b of the second imaging device 2b transmits the second additional information to the communication unit 30 of the information processing device 3 in step S68. The communication unit 30 receives the second additional information from the communication unit 13b in step S69.

[0100] The imaging system 1 integrates the first model information and the second model information by the processing described in steps S43 to S54, for example, with reference to Figure 9. The model integration unit 45 of the information processing device 3 stores the calculated integrated model information in the storage unit 31, for example. In step S70, the model integration unit 45 checks (determines) the consistency of the calculated integrated model information. If the model integration unit 45 determines that the integrated model information is not consistent, it performs the following processing, for example.

[0101] In step S71, the control unit 33 of the information processing device 3 controls the communication unit 30 and sends a command (request signal) to the communication unit 13a of the first imaging device 2a requesting the transmission of the first interpolation information. The first interpolation information is, for example, information used for interpolating integrated model information and includes information on the overlapping region between the shape shown by the first model information and the shape shown by the second model information. In step S72, the communication unit 13a of the first imaging device 2a receives the command from step S68. In step S73, the information calculation unit 12a of the first imaging device 2a extracts the first interpolation information from the first model information. In step S74, the communication unit 13a of the first imaging device 2a transmits the first interpolation information to the communication unit 30 of the information processing device 3. In step S75, the communication unit 30 receives the first interpolation information from the communication unit 13a.

[0102] In step S76, following step S71, the control unit 33 of the information processing device 3 controls the communication unit 30 and sends a command (request signal) to the communication unit 13b of the second imaging device 2b requesting the transmission of second interpolation information. The second interpolation information is, for example, information used for interpolating integrated model information and includes information on the overlapping region between the shape shown by the first model information and the shape shown by the second model information. In step S77, the communication unit 13b of the second imaging device 2b receives the command from step S76. In step S78, the information calculation unit 12b of the second imaging device 2b extracts the second interpolation information from the second model information. In step S79, the communication unit 13b of the second imaging device 2b transmits the second interpolation information to the communication unit 30 of the information processing device 3. In step S80, the communication unit 30 receives the second interpolation information from the communication unit 13b.

[0103] In step S81, the model integration unit 45 of the information processing device 3 performs interpolation processing on the integrated model information using the first interpolation information and the second interpolation information. In step S82, the model integration unit 45 updates the integrated model information stored in the storage unit 31 with the integrated model information after interpolation processing.

[0104] Furthermore, for example, the model integration unit 45 generates integrated model information using information obtained through the sharing of model information (e.g., first model information, second model information) or feature point data (e.g., first feature point data, second feature point data) between multiple imaging devices (e.g., information sharing via bidirectional communication, one-way communication) (e.g., additional information, specified information).

[0105] [Fifth Embodiment] Next, a fifth embodiment will be described. In this embodiment, components similar to those in the embodiments described above are denoted by the same reference numerals, and their descriptions are simplified or omitted. Figure 11 is a block diagram showing the imaging system 1 according to this embodiment.

[0106] The imaging system 1 according to this embodiment comprises a first imaging device 2a, a second imaging device 2b, and an information processing device 3. The first imaging device 2a includes, for example, a model integration unit 35 (hereinafter referred to as the model integration unit 35a) similar to that in the second embodiment. The model integration unit 35a integrates the first model information calculated by the information calculation unit 12a and the second model information received by the communication unit 13a from the second imaging device 2b. The communication unit 13a transmits the integrated model information calculated by the model integration unit 35a to the communication unit 30 of the information processing device 3.

[0107] Figure 12 is a diagram showing the operation of the imaging system 1 according to this embodiment. In Figure 12, the same reference numerals are used for processes similar to those in Figure 9, etc., and their explanations are simplified or omitted. The imaging system 1 calculates first additional information used for model integration through the processes of steps S31 to S40.

[0108] In step S85, the first imaging device 2a determines and sets a first area to be handled by the first model information and a second area to be handled by the second model information from the second imaging device 2b, from the integrated model information. For example, the model integration unit 35a selects each area so that the first and second areas do not overlap.

[0109] In step S86, the control unit 28 of the first imaging device 2a controls the communication unit 13a and sends a command to the communication unit 13b of the second imaging device 2b requesting the transmission of second model information in the second assigned area. In step S87, the communication unit 13b of the second imaging device 2b receives the command from step S86. In step S88, the information calculation unit 12b of the second imaging device 2b extracts information for the second assigned area specified by the first imaging device 2a from the second model information. In step S89, the communication unit 13b of the second imaging device 2b transmits the information extracted by the information calculation unit 12b in step S86 (extracted information) to the communication unit 13a of the first imaging device 2a.

[0110] In step S90, the information calculation unit 12a of the first imaging device 2a extracts the first model information for the first assigned area. In step S91, the communication unit 13a of the first imaging device 2a receives the extracted information (e.g., the second model information for the second assigned area) from the communication unit 13b. In step S92, the model integration unit 35a of the first imaging device 2a integrates the first model information for the first assigned area and the second model information for the second assigned area to calculate integrated model information. In step S93, the communication unit 13a of the first imaging device 2a transmits the integrated model information to the communication unit 30 of the information processing device 3. In step S94, the communication unit 30 receives the integrated model information from the communication unit 13a. In step S95, the rendering processing unit 32 of the information processing device 3 performs rendering processing using the integrated model information.

[0111] Furthermore, for example, the model integration unit 35a of the first imaging device 2a generates integrated model information using information obtained through the sharing of model information (e.g., first model information, second model information) or feature point data (e.g., first feature point data, second feature point data) between multiple imaging devices (e.g., information sharing via bidirectional communication, one-way communication) (e.g., additional information, specified information).

[0112] [Sixth Embodiment] Next, a sixth embodiment will be described. In this embodiment, components similar to those in the embodiments described above are denoted by the same reference numerals, and their descriptions are simplified or omitted. Figure 13 is a block diagram showing the imaging system 1 according to this embodiment.

[0113] In this embodiment, the first imaging device 2a and the second imaging device 2b communicate information used for model integration with each other and each calculate integrated model information. The first imaging device 2a calculates integrated model information by, for example, the process described with reference to Figure 12. The second imaging device 2b also calculates integrated model information by, for example, performing the same process as the first imaging device 2a.

[0114] The information processing device 3 receives integrated model information from at least one of the first imaging device 2a and the second imaging device 2b, and performs rendering processing using the received integrated model information. For example, the communication unit 30 of the information processing device 3 sends a command to the communication unit 13a of the first imaging device 2a requesting the transmission of integrated model information. The control unit 28 of the first imaging device 2a controls the communication unit 13a to transmit information of the items specified in the command from the integrated model information to the communication unit 30 of the information processing device 3. The information processing device 3 can similarly cause the second imaging device 2b to transmit integrated model information.

[0115] The information processing device 3 may, for example, monitor the load information of each part of the imaging system 1 (e.g., the ratio of load to processing power) and assign the model integration processing to the processing unit with a relatively small load in the imaging system 1. For example, the control unit 28 of the first imaging device 2a generates load information for the first imaging device 2a. The load information for the first imaging device 2a includes, for example, the load information of the information calculation unit 12a and the load information of the model integration unit 35a. The communication unit 13a of the first imaging device 2a transmits the load information for the first imaging device 2a to the communication unit 30 of the information processing device 3. Similarly, the second imaging device 2b transmits the load information for the second imaging device 2b to the communication unit 30 of the information processing device 3. The control unit 33 of the information processing device 3 compares the load of each part based on the received load information and assigns the processing (model integration processing) to the processing unit with a relatively small load, either by having only one imaging device execute the processing or by partially distributing the processing (model integration processing) among multiple imaging devices.

[0116] For example, if the control unit 33 determines that the load on the model integration unit 35a of the first imaging device 2a is greater than the load on the model integration unit 35b of the second imaging device 2b, it may restrict at least a part of the model integration performed by the first imaging device 2a and have the second imaging device 2b perform the restricted processing. For example, the control unit 33 may prohibit the calculation of first additional information by the first imaging device 2a and have the second imaging device 2b transmit second additional information to the first imaging device 2a, and the first imaging device 2a may perform model integration using the second additional information. Alternatively, for example, the control unit 33 may prohibit the determination of the assigned area by the first imaging device 2a and have the second imaging device 2b transmit assigned area information defining the assigned area to the first imaging device 2a, and the first imaging device 2a may perform model integration using the assigned area information. Alternatively, for example, the control unit 33 may prohibit model integration by the first imaging device 2a and receive integrated model information from the second imaging device 2b.

[0117] Furthermore, if the control unit 33 determines that the load on the information processing device 3 is smaller than that of either the first imaging device 2a or the second imaging device 2b, it may restrict at least a portion of the model integration process of at least one of the first imaging device 2a or the second imaging device 2b, and have the model integration unit 45 execute the restricted process. For example, the control unit 33 may, for instance, prohibit the determination of the assigned area by the first imaging device 2a and the second imaging device 2b, have the model integration unit 45 execute the determination of the assigned area, and have the model integration unit 45 transmit the assigned area information to the first imaging device 2a and the second imaging device 2b, respectively. Alternatively, the control unit 33 may prohibit the model integration process by the first imaging device 2a and the second imaging device 2b, and have the model integration unit 45 of the information processing device 3 execute the model integration process.

[0118] Furthermore, as with the control unit 33 described above, the load control unit (load monitoring unit) that performs at least one of the control and monitoring of the load of the imaging system 1 may be provided in the first imaging device 2a (e.g., control unit 28) or in the second imaging device 2b (e.g., control unit 28). Also, the load control unit (load monitoring unit) may be provided outside the imaging system 1 or not provided in the imaging system 1.

[0119] Furthermore, for example, the model integration unit 35a of the first imaging device 2a (or / and the model integration unit 35b of the second imaging device 2b) generates integrated model information using information obtained through the sharing of model information (e.g., first model information, second model information) or feature point data (e.g., first feature point data, second feature point data) between multiple imaging devices (e.g., information sharing by bidirectional communication, one-way communication) (e.g., additional information, specified information).

[0120] [Seventh Embodiment] Next, a seventh embodiment will be described. In this embodiment, components similar to those in the embodiments described above are denoted by the same reference numerals, and their descriptions are simplified or omitted. Figure 14 is a block diagram showing the imaging system 1 according to this embodiment.

[0121] The imaging system 1 according to this embodiment comprises a first imaging device 2a, a second imaging device 2b, and an information processing device 3. The first imaging device 2a and the second imaging device 2b are each capable of calculating integrated model information. In this embodiment, the information processing device 3 does not include a model integration unit. The information processing device 3 receives integrated model information from at least one of the first imaging device 2a and the second imaging device 2b, and performs rendering processing using the received integrated model information.

[0122] Figure 15 is a diagram showing the operation of the imaging system 1 according to this embodiment. In the following description, processes equivalent to those in Figure 10, etc., are denoted by the same reference numerals, and their descriptions are omitted or simplified. The first imaging device 2a calculates first model information (step S31) and calculates first feature point data (step S32). The second imaging device 2b calculates second model information (step S33) and calculates first feature point data (step S34). The communication unit 13a of the first imaging device 2a transmits the first feature point data to the communication unit 13b of the second imaging device 2b (step S60), and the communication unit 13b receives the first feature point data (step S61). The communication unit 13b transmits the second feature point data to the communication unit 13a of the first imaging device 2a (step S62), and the communication unit 13a receives the second feature point data (step S63).

[0123] The information calculation unit 12a of the first imaging device 2a calculates first additional information using the first feature point data and the second feature point data (step S64). The information calculation unit 12b of the second imaging device 2b calculates second additional information using the first feature point data and the second feature point data (step S67). The communication unit 13a of the first imaging device 2a transmits the first additional information to the communication unit 13b of the second imaging device 2b (step S101), and the communication unit 13b receives the first additional information (step S102). The communication unit 13b transmits the second additional information to the communication unit 13a of the first imaging device 2a (step S103), and the communication unit 13a receives the second additional information (step S104).

[0124] The model integration unit 35a of the first imaging device 2a determines the area to be handled using the first additional information and the second additional information (step S105). The information calculation unit 12a of the first imaging device 2a extracts the first model information for the first area to be handled (step S106). The model integration unit 35b of the second imaging device 2b determines the area to be handled using the first additional information and the second additional information (step S107). The information calculation unit 12b of the second imaging device 2b extracts the second model information for the second area to be handled (step S108).

[0125] The communication unit 13a of the first imaging device 2a transmits the first model information for the first assigned area to the communication unit 13b of the second imaging device 2b (step S109), and the communication unit 13b receives the first model information for the first assigned area (step S110). The communication unit 13b transmits the second model information for the second assigned area to the communication unit 13a of the first imaging device 2a (step S111), and the communication unit 13a receives the second model information for the second assigned area (step S112).

[0126] The model integration unit 35a of the first imaging device 2a performs model integration using the first model information of the first assigned area and the second model information of the second assigned area (step S113). Then, the model integration unit 35a checks the integrated model information (first integrated model information) (step S114). The model integration unit 35b of the second imaging device 2b performs model integration using the first model information of the first assigned area and the second model information of the second assigned area (step S115). Then, the model integration unit 3ba checks the integrated model information (second integrated model information) (step S116).

[0127] The model integration unit 35a of the first imaging device 2a and the model integration unit 35b of the second imaging device 2b perform the following processing, for example, when they determine that the integrated model information is inconsistent: The information calculation unit 12a of the first imaging device 2a extracts first interpolation information (step S117). The information calculation unit 12b of the second imaging device 2b extracts second interpolation information (step S118).

[0128] The communication unit 13a of the first imaging device 2a transmits the first interpolation information to the communication unit 13b of the second imaging device 2b (step S119), and the communication unit 13b receives the first interpolation information (step S120). The communication unit 13b transmits the second interpolation information to the communication unit 13a of the first imaging device 2a (step S121), and the communication unit 13a receives the second interpolation information (step S122).

[0129] The model integration unit 35a of the first imaging device 2a performs interpolation processing using the first interpolation information and the second interpolation information (step S123). The model integration unit 35a updates the integrated model information stored in the storage unit 27a with the integrated model information after interpolation processing (step S124). The model integration unit 35b of the second imaging device 2b performs interpolation processing using the first interpolation information and the second interpolation information (step S125). The model integration unit 35b updates the integrated model information stored in the storage unit 27b with the integrated model information after interpolation processing (step S126).

[0130] The information processing device 3 receives information from the first imaging device 2a. For example, the communication unit 30 of the information processing device 3 sends a command to the communication unit 13a of the first imaging device 2a requesting the transmission of information used for rendering (step S130), and the communication unit 13a receives the command in step S130 (step S131). The information calculation unit 12a of the first imaging device 2a extracts the information specified in the command in step S130 from, for example, model information (first model information or integrated model information) (step S134). The communication unit 13a of the first imaging device 2a transmits the information extracted in step S134 to the communication unit 30 of the information processing device 3 (step S135), and the communication unit 30 receives the information transmitted in step S134 (step S136).

[0131] The information processing device 3 receives information from the second imaging device 2b as needed. For example, when using information held by the second imaging device 2b for rendering processing, the communication unit 30 of the information processing device 3 sends a command to the communication unit 13b of the second imaging device 2b requesting the transmission of information to be used for rendering processing (step S132), and the communication unit 13b receives the command in step S132 (step S133). The information calculation unit 12b of the second imaging device 2b extracts the information specified in the command in step S132 from, for example, model information (second model information or integrated model information) (step S137). The communication unit 13b of the second imaging device 2b transmits the information extracted in step S137 to the communication unit 30 of the information processing device 3 (step S138), and the communication unit 30 receives the information transmitted in step S138 (step S139).

[0132] The rendering processing unit 32 of the information processing device 3 performs rendering processing using, for example, the information received from the first imaging device 2a and the information received from the second imaging device 2b. Note that if the information processing device 3 does not use the information held by the second imaging device 2b for rendering processing, it does not need to receive information from the second imaging device 2b. For example, the communication unit 30 of the information processing device 3 does not need to send a command to the communication unit 13b of the second imaging device 2b requesting the transmission of information.

[0133] Furthermore, for example, the model integration unit 35a of the first imaging device 2a generates integrated model information (first integrated model information) using information obtained through the sharing of model information (e.g., first model information, second model information) or feature point data (e.g., first feature point data, second feature point data) among multiple imaging devices (e.g., information sharing via bidirectional communication, one-way communication) (e.g., additional information, specified information). For example, the model integration unit 35b of the second imaging device 2b generates integrated model information (second integrated model information) using information obtained through the sharing of model information (e.g., first model information, second model information) or feature point data (e.g., first feature point data, second feature point data) among multiple imaging devices (e.g., information sharing via bidirectional communication, one-way communication) (e.g., additional information, specified information). Thus, for example, when multiple imaging devices image the same object OB and generate integrated model information in each imaging device, the multiple imaging devices (e.g., first imaging device 2a, second imaging device 2b) may be configured to distribute the model integration of the entire object OB in each imaging device and generate integrated model information (e.g., first integrated model information, second integrated model information). In this case, for example, the integrated model information generated by distributed processing in the imaging devices is partial integrated model information of the object OB. In such a case, for example, the first integrated model information generated by distributed processing may be a partial integrated model corresponding to 50% of the entire area of ​​the object OB, and the first integrated model information and the second integrated model information may be configured to share the integrated model information of the entire object OB. Then, in response to requests from playback devices such as the information processing device 3 or a rendering device, the imaging device transmits the information necessary for rendering (e.g., first integrated model information, second integrated model information) to the information processing device 3 or the playback device.

[0134] [Eighth Embodiment] Next, an eighth embodiment will be described. In this embodiment, components similar to those in the embodiments described above are denoted by the same reference numerals, and their descriptions are simplified or omitted. Figure 16 is a block diagram showing the imaging system 1 according to this embodiment.

[0135] In this embodiment, the first imaging device 2a, the second imaging device 2b, and the information processing device 3 do not have the model integration unit described above. The information processing device 3, for example, performs a first rendering process using information received from the first imaging device 2a (e.g., integrated model information), and then performs a second rendering process using information received from the second imaging device 2b (e.g., integrated model information). The rendering processing unit 32, for example, generates a first estimated image of the object OB viewed from a first viewpoint in the first rendering process. The rendering processing unit 32, for example, generates a second estimated image of the object OB viewed from a second viewpoint in the second rendering process. The information processing device 3, for example, requests the second imaging device 2b to transmit information if the information held by the first imaging device 2a alone results in a model omission when generating the second estimated image. The communication unit 30 of the information processing device 3 may, for example, sequentially receive information used in the second rendering process during the period in which the first rendering process is performed.

[0136] In such a configuration, for example, in order for the information processing device 3 to obtain the information necessary for rendering the information processing device 3, the imaging system 1 in this embodiment further includes an information processing device (model integration processing device) which includes a communication unit that can receive information such as model information or additional information and transmit the calculated integrated model information to an external device, and a model integration unit that performs model integration processing.

[0137] Furthermore, at least one of the first imaging device 2a, the second imaging device 2b, and the information processing device 3 may be equipped with a model integration unit. For example, the second imaging device 2b may be equipped with a model integration unit, while the first imaging device 2a and the information processing device 3 may not be equipped with a model integration unit. In this case, for example, the information processing device 3 may perform the first rendering process described above using the information received from the first imaging device 2a. The second imaging device 2b may also perform at least a part of the model integration process during the period in which the first rendering process is performed. The information processing device 3 may also perform a second rendering process after the first rendering process using the integrated model information received from the second imaging device 2b.

[0138] [Ninth Embodiment] Next, a ninth embodiment will be described. In this embodiment, components similar to those in the above-described embodiments are denoted by the same reference numerals, and their descriptions are simplified or omitted. Figure 17 is a block diagram showing the imaging system 1 according to this embodiment.

[0139] In this embodiment, the communication unit 13 of the imaging device 2 can receive image data of the object OB from an external device (e.g., imaging device 50 as a second imaging device). The imaging device 50 may be part of the imaging system 1 or it may be an external device to the imaging system 1. The imaging device 50 does not have a distance measuring unit like the imaging device 2. In this case, the imaging device 50 transmits the image with parallax (parallax image) from the image captured by the imaging unit 51 to the imaging device 2. The imaging device 2 then calculates the depth information of the imaging device 50 using the received parallax image. The imaging device 50 does not calculate the model information described above. The imaging device 50 may be, for example, a general-purpose camera, a high-function mobile phone such as a smartphone, or a tablet.

[0140] The imaging device 50 includes, for example, an imaging unit 51, an input unit 52, a storage unit 53, a display unit 54, a communication unit 55, and a control unit 56. The imaging unit 51 includes, for example, a CCD image sensor or a CMOS image sensor and is capable of imaging an object OB. The storage unit 53 includes, for example, a non-volatile memory and stores image data and various setting information captured by the imaging unit 51. The input unit 52 receives, for example, commands from the user. The display unit 54 displays, for example, image data stored in the storage unit 53 or images indicated by the various setting information. The communication unit 55 can communicate with the communication unit 13 of the imaging device 2 by wire or wireless. The communication unit 55 transmits, for example, image data of the object OB captured by the imaging unit 51 to the communication unit 13 of the imaging device 2. The control unit 56 controls each part of the imaging device 50.

[0141] As described in the above embodiment, the information calculation unit 12 of the imaging device 2 calculates first model information using the detection result of the detection unit 11. The information calculation unit 12 of the imaging device 2 also calculates second model information based on, for example, the image captured by the imaging unit 15 and / or the image received from the imaging device 50 (image data of the target object OB). The information calculation unit 12 does not have to calculate the second model information; for example, the image received from the imaging device 50 may be used for interpolation processing. For example, the imaging device 2 may detect first depth information using the distance measuring unit 16, calculate second depth information based on the image captured by the imaging unit 15 and the image received from the imaging device 50, and interpolate the first depth information using the second depth information.

[0142] The communication unit 55 of the imaging device 50 can communicate with the communication unit 30 of the information processing device 3 by wire or wireless connection. The communication unit 55 transmits, for example, rendering processing setting information (e.g., viewpoint information of the estimated image) to the communication unit 30 of the information processing device 3. The rendering processing unit 32 of the information processing device 3 performs rendering processing using, for example, the model information received from the communication unit 13 of the imaging device 2 and the rendering processing setting information received from the communication unit 55 of the imaging device 30. The communication unit 30 of the information processing device 30 transmits, for example, the estimated image data generated by the rendering process to the communication unit 55 of the imaging device 50. The display unit 54 of the imaging device 50 displays the estimated image based on the estimated image data from the information processing device 30.

[0143] Note that the communication unit 55 of the imaging device 50 does not need to communicate with the communication unit 30 of the information processing device 3. For example, the communication unit 55 may transmit rendering processing setting information to the communication unit 13 of the imaging device 2. The communication unit 13 may transmit the setting information received from the communication unit 55 of the imaging device 50 to the communication unit 30 of the information processing device 3. The communication unit 13 may also receive estimated image data indicating the result of the rendering process from the communication unit 30 of the information processing device 3. The communication unit 13 may transmit the estimated image data received from the communication unit 30 of the information processing device 3 to the communication unit 55 of the imaging device 50. The display unit 54 of the imaging device 50 may display the estimated image based on the estimated image data from the imaging device 2.

[0144] The imaging system 1 or imaging device 2 in each of the embodiments described above may be, for example, at least part of a video content production system or production support system including CG, or at least part of a blood vessel or biological tissue visualization imaging system, a biological microscope system, a defect detection system, or an object measurement system. Furthermore, the imaging system 1 or imaging device 2 may be at least part of a detection system that detects, for example, the intrusion of objects, animals, etc., into a field of view, or the movement of objects, animals, etc., within a field of view. This detection system may be, for example, a security system, a nursing care system, a monitoring system, a traffic light system, or an information acquisition system that acquires statistical data such as traffic volume. Also, the imaging system 1 or imaging device 2 may be at least part of a simulator. This simulator may generate estimated images of objects (e.g., furniture) or animals (e.g., people, pets) placed in a space (e.g., a room) modeled by the imaging device 2. Alternatively, this simulator may generate estimated images of objects or animals modeled by the imaging device 2 placed in a predetermined space (e.g., a room).

[0145] Furthermore, according to each of the embodiments described above, the imaging system in this embodiment includes a first imaging device (2, 2a) comprising: a first detection unit including a first imaging unit for imaging an object and a first distance measuring unit for detecting the distance from each point on the surface of the object; and a first information calculation unit that calculates first model information including at least one of shape information and texture information of the object using the detection results of the first detection unit, and generates first feature point data using the first model information; and a second imaging device including a second imaging unit for imaging an object and a second distance measuring unit for detecting the distance from each point on the surface of the object. The configuration may also include: a second imaging device (2, 2b) comprising: a detection unit; a second information calculation unit that calculates second model information including at least one of shape information and texture information of an object using the detection results of the second detection unit, and generates second feature point data using the second model information; and a model integration unit (35, 45) that integrates the first model information and the second model information using the first feature point data and the second feature point data obtained by data communication between the first imaging device and the second imaging device to generate integrated model information of an object. In such a configuration, the model integration unit may be provided in at least the first imaging device, the second imaging device, or the information processing device, or it may be provided in another external terminal (model integration processing device).

[0146] It should be noted that the technical scope of the present invention is not limited to the embodiments or modifications described above. For example, one or more of the requirements described in the embodiments or modifications described above may be omitted. Also, the requirements described in the embodiments or modifications described above can be combined as appropriate.

[0147] The imaging system 1 may have three or more imaging devices. For example, the imaging system 1 may have four imaging devices, two of which may have model integration units, while the remaining two may not have model integration units. For example, an imaging device equipped with a model integration unit may receive model information from an imaging device that does not have a model integration unit and perform model integration. In this case, first integrated model information and second integrated model information are calculated, and the information processing device 3 may integrate the first integrated model information and the second integrated model information. The imaging device in this embodiment may be a wearable terminal, and may have a wearable part that can be attached to the arm or head, and an input part that allows voice input, input buttons, or touch operation on the main body. [Explanation of Symbols]

[0148] 1...Imaging system, 2...Imaging device, 3...Information processing device, 11...Detection unit, 12...Information calculation unit, 13...Communication unit, 15...Imaging unit, 16...Distance measuring unit, 18...Image sensor, 19...Illumination unit, 21...Image sensor, 35...Model Integration Department, OB...Target Object

Claims

1. A first detection device that detects an object from a first position, calculates first model information of the detected object, detects a first feature point which is a part of the shape indicated by the calculated first model information that is distinguishable from other parts, and calculates first feature point data which indicates the position of the detected first feature point and the feature quantity of the feature point, A second detection device that detects the object from a second position different from the first position, calculates second model information of the detected object, detects a second feature point which is a part of the shape indicated by the calculated second model information that can be distinguished from other parts, and calculates second feature point data which indicates the position of the detected second feature point and the feature quantity of the feature point. An information processing device that generates integrated model information by integrating the first model information and the second model information, Equipped with, The first detection device, the second detection device, and the information processing device are capable of communicating with each other. The first detection device acquires the second feature point data from the second detection device, and calculates first additional information, including the positional relationship between the shape indicated by the first model information and the information indicated by the second model information, from the first feature point data and the acquired second feature point data. The second detection device acquires the first feature point data from the first detection device, and calculates second additional information, including the positional relationship between the shape indicated by the first model information and the shape indicated by the second model information, from the acquired first feature point data and the second feature point data. The information processing device acquires the first additional information from the first detection device and the second additional information from the second detection device, and determines the areas to be handled by the first model information from the first detection device and the second model information from the second detection device from the acquired first additional information and second additional information such that the first area to be handled by the first model information from the first detection device and the second area to be handled by the second model information from the second detection device do not overlap, and transmits a command to the first detection device requesting the first designated information, which is the first model information in the first area to be handled, and transmits a command to the second detection device requesting the second designated information, which is the second model information in the second area to be handled. The first detection device extracts the first designated information from the first model information based on a command from the information processing device, and transmits the extracted first designated information to the information processing device. The second detection device extracts the second designation information from the second model information based on the designation from the information processing device, and transmits the extracted second designation information to the information processing device. The information processing device is a system that integrates the first designated information received from the first detection device and the second designated information received from the second detection device to generate the integrated model information.

2. The first additional information includes information on the timing at which the object was detected by the first detection device, The second additional information includes information about the timing at which the object was detected by the second detection device. The information processing device integrates the first model information and the second model information based on at least two detection results in which the timing of detection of the object is relatively close, using the timing information included in the first additional information and the timing information included in the second additional information, according to claim 1.

3. The first detection device shall not transmit model information in a range not specified in the first designated information. The system according to claim 1, wherein the second detection device does not transmit model information in a range not specified in the second designated information.

Images