Image processing apparatus, image processing method, and storage medium

Abstract

Provided is an image processing apparatus, an image processing method, and a storage medium. A processor of an image processing apparatus performs the following processing: acquires a three-dimensional ranging sensor system three-dimensional coordinate that can determine the positions of a plurality of measurement points and a three-dimensional ranging sensor system three-dimensional coordinate that is defined by a three-dimensional coordinate system that is appropriate for a three-dimensional ranging sensor, acquires a camera system three-dimensional coordinate from the three-dimensional ranging sensor system three-dimensional coordinate, converts the camera system three-dimensional coordinate into a camera system two-dimensional coordinate that can determine positions within a captured image, converts the camera system three-dimensional coordinate into a display system two-dimensional coordinate that can determine positions within a screen, and allocates pixels that constitute the captured image to interpolation positions within the screen that are determined using an interpolation method that uses the camera system two-dimensional coordinate and the display system two-dimensional coordinate.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a storage medium. Background Technology

[0002] Japanese Patent Application Publication No. 2012-220471 discloses a development diagram generation apparatus. The development diagram generation apparatus described in Japanese Patent Application Publication No. 2012-220471 comprises: a storage mechanism for storing an image obtained by photographing the wall of a tunnel and measurement point data having coordinate values ​​and reflection intensity values ​​of multiple measurement points on the wall obtained by laser scanning measurement of the wall; a conversion mechanism for performing coordinate conversion of the multiple measurement points on the wall to form a development diagram of the wall; a comparison mechanism for aligning the image with the coordinate-converted multiple measurement points based on the reflection intensity values; a displacement shape generation mechanism for generating a displacement shape by assigning a concave-convex shape to a coordinate value reflecting a direction orthogonal to the development plane of the wall, based on the coordinates of the coordinate-converted multiple measurement points; and a drawing mechanism for drawing a pattern of the aligned image on the displacement shape.

[0003] Japanese Patent Application Publication No. 2017-106749 discloses a point cloud data acquisition system, which includes a first electronic device group having a first electronic device and one or more movable second electronic devices, and acquires point cloud data with depth information up to each point on the surface of a subject in pixels.

[0004] In the point cloud data acquisition system described in Japanese Patent Application Publication No. 2017-106749, the second electronic device includes: a plurality of markers, which are defined markers having a second coordinate system and having a plurality of visual features, and the plurality of markers are respectively set at linearly independent positions in the second coordinate system; and a three-dimensional measurement unit that measures the point cloud data of the subject according to the second coordinate system. Furthermore, in the point cloud data acquisition system described in Japanese Patent Application Publication No. 2017-106749, the first electronic device includes a measuring unit that has a first coordinate system and measures the position information of the subject according to the first coordinate system. The point cloud data acquisition system also includes: a marker position information calculation unit that has a reference coordinate system and calculates the coordinate values ​​of multiple markers of the second electronic device in the first coordinate system, i.e., the position information of the second electronic device, based on the data measured by the position information measuring unit; a point cloud data coordinate value conversion unit that converts the coordinate values ​​of each point in the point cloud data measured by the three-dimensional measuring unit of the second electronic device in the second coordinate system into coordinate values ​​in the reference coordinate system based on the position information of the second electronic device; and a synthetic point cloud data creation unit that creates a synthetic point cloud data from multiple point cloud data measured by the three-dimensional measuring unit of the second electronic device based on the coordinate values ​​in the reference coordinate system converted by the point cloud data coordinate value conversion unit.

[0005] Japanese Patent Application Publication No. 2018-025551 discloses a technology comprising: a first electronic device, a depth camera that measures point cloud data of a subject according to a depth camera coordinate system; and a second electronic device, a non-depth camera that acquires two-dimensional image data of a subject according to a non-depth camera coordinate system. In this technology, a corresponding association is established between the two-dimensional image data of the subject acquired by the non-depth camera and the point cloud data obtained by converting the coordinate values ​​of each point on the surface of the subject measured by the depth camera in the depth camera coordinate system into coordinate values ​​in the non-depth camera coordinate system, thereby converting the two-dimensional image data of the subject into point cloud data.

[0006] Japanese Patent No. 4543820 discloses a three-dimensional data processing apparatus comprising: an intermediate interpolation mechanism for interpolating auxiliary intermediate points of each line data of a three-dimensional vector data containing multiple line data; a TIN generation mechanism for forming an irregular triangular network based on the three-dimensional coordinate values ​​of the points of each line data describing the three-dimensional vector data and the auxiliary intermediate points, and generating TIN (Triangulated Irregular Network) data for each triangle; and a grid coordinate calculation mechanism for applying a grid with a specified grid interval to the TIN data and calculating the coordinate values ​​of each grid point based on the TIN data, and outputting grid data representing the three-dimensional coordinates for each grid point. The grid coordinate calculation mechanism includes a mechanism for searching for TINs contained in the TIN data and a coordinate value calculation mechanism for calculating the coordinate values ​​of the grid points contained in the searched TINs. In the three-dimensional data processing apparatus described in Japanese Patent No. 4543820, a coordinate value calculation mechanism determines the maximum grid range contained in a quadrilateral circumscribed with a TIN, searches for grid points contained in the TIN within the maximum grid range, and calculates the coordinate values ​​of each grid point searched based on the coordinate values ​​of the three vertices of the TIN.

[0007] Non-Patent Document 1 discloses a method for combining measured temperature data with the three-dimensional geometry of a building. The method described in Non-Patent Document 1 utilizes a function that virtually represents the energy efficiency and environmental load of an actual building. Furthermore, the method provides visual information using a hybrid LiDAR (Light Detection and Ranging) system. This facilitates the renovation of buildings.

[0008] In the method described in Non-Patent Document 1, the 3D temperature model simultaneously maintains the geometric point cloud data of the building and the temperature data (temperature and temperature color information generated based on the temperature) of each point. LiDAR cannot collect geometric data from transparent objects, so in the case of window glass, it is necessary to create virtual temperature vertices. Therefore, in the method described in Non-Patent Document 1, an algorithm for determining windows is used. The framework of the service proposed in Non-Patent Document 1 consists of the following components: (1) A hybrid 3D LiDAR system that simultaneously collects point clouds and temperatures from the exterior walls of an actual building. (2) Automatic synthesis of the collected point cloud and temperature data. (3) A window detection algorithm to compensate for the inability of LiDAR to detect window glass. (4) Drawing to a GUI. (5) A web-based floor plan layout program for determining repairs, etc.

[0009] Non-Patent Document 1 describes that a LiDAR and IR (Infrared Rays) camera is fixed to the base of a PTU (Pan and Tilt Unit) (see reference). Figure 5 The distortion aberration of the IR camera was pre-calibrated in the photographic system. Furthermore, Non-Patent Document 1 describes the use of a black and white inspection plate (see reference). Figure 7 Furthermore, Non-Patent Document 1 describes the following: in order to synthesize LiDAR-based point clouds and images obtained through an IR camera, it is necessary to calibrate internal and external parameters (see reference). Figure 6 Internal parameters refer to focal length, principal point of the lens, skew distortion coefficient, and distortion aberration coefficient, while external parameters refer to the rotation and parallel translation matrices. Summary of the Invention

[0010] One embodiment of the present invention provides an image processing apparatus, image processing method, and program that can facilitate the high-precision creation of images within a frame.

[0011] means for solving technical problems

[0012] The first aspect of the present invention relates to an image processing apparatus comprising: a processor; and a memory connected to or built into the processor, wherein the positional relationship between a three-dimensional ranging sensor of the image processing apparatus and a camera device with a sampling resolution higher than that of the three-dimensional ranging sensor is known, wherein the processor performs the following processing: based on the ranging results of the three-dimensional ranging sensor for multiple measuring points, obtaining three-dimensional coordinates of the three-dimensional ranging sensor system that can determine the positions of the multiple measuring points and defined by a three-dimensional coordinate system suitable for the three-dimensional ranging sensor; based on the three-dimensional coordinates of the three-dimensional ranging sensor system, obtaining three-dimensional coordinates of the camera device system defined by a three-dimensional coordinate system suitable for the camera device; converting the three-dimensional coordinates of the camera device system into two-dimensional coordinates of the camera device system that can determine the positions within a photographic image obtained by the camera device; converting the three-dimensional coordinates of the camera device system into two-dimensional coordinates of the display system that can determine the positions within a frame; and allocating the pixels constituting the photographic image to the frame at interpolated positions determined by an interpolation method using the two-dimensional coordinates of the camera device system and the two-dimensional coordinates of the display system.

[0013] The second aspect of the present invention is the image processing apparatus of the first aspect, wherein the processor generates polygonal image patches based on the three-dimensional coordinates of the camera device system, and the three-dimensional coordinates of the camera device system define the positions of the intersection points of the polygonal image patches.

[0014] The third aspect of the technology of the present invention is the image processing apparatus of the second aspect, wherein the interpolation position is the position within the image corresponding to a position other than the intersection of polygonal image blocks.

[0015] The fourth aspect of the present invention is the image processing apparatus involved in the second or third aspect, wherein the processor performs the following processing: further allocating the pixels of the multiple pixels contained in the photographed image to the positions corresponding to the intersection points of the polygonal image blocks within the frame, thereby creating a three-dimensional image.

[0016] The fifth aspect of the technology of the present invention is an image processing apparatus involved in any of the second to fourth aspects, wherein the polygonal image blocks are defined by a triangular mesh or a quadrilateral mesh.

[0017] The sixth aspect of the present invention is an image processing apparatus according to any one of the first to fifth aspects, wherein the processor performs the following processing: converting the three-dimensional coordinates of the three-dimensional ranging sensor system into the three-dimensional coordinates of the camera device system, thereby obtaining the three-dimensional coordinates of the camera device system.

[0018] The seventh aspect of the present invention relates to an image processing apparatus according to any one of the first to sixth aspects, wherein the processor performs the following processing: calculating to obtain the three-dimensional coordinates of the camera device system based on feature points of a subject contained as an image between multiple frames of images, the multiple frames of images being obtained by the camera device capturing the subject from different positions.

[0019] The eighth aspect of the present invention is an image processing apparatus according to any one of the first to seventh aspects, wherein the position within the screen is the position within the screen of the display.

[0020] The ninth aspect of the present invention is an image processing apparatus according to any one of the first to eighth aspects, wherein the memory stores corresponding association information obtained by establishing a corresponding association between the three-dimensional coordinates of the three-dimensional ranging sensor system and the two-dimensional coordinates of the camera device system.

[0021] The tenth aspect of the present invention is the image processing apparatus of the ninth aspect, wherein the processor performs the following processing: assigning pixels constituting a camera image to interpolation positions with reference to corresponding association information.

[0022] The eleventh aspect of the present invention is an image processing method comprising the following steps: Given that the positional relationship between a three-dimensional ranging sensor and a camera device with a higher sampling resolution than the three-dimensional ranging sensor is known, based on the ranging results of the three-dimensional ranging sensor for multiple measurement points, obtaining three-dimensional coordinates of the three-dimensional ranging sensor system that can determine the positions of the multiple measurement points, and defined by a three-dimensional coordinate system suitable for the three-dimensional ranging sensor; obtaining three-dimensional coordinates of the camera device system defined by a three-dimensional coordinate system suitable for the camera device based on the three-dimensional coordinates of the three-dimensional ranging sensor system; converting the three-dimensional coordinates of the camera device system into two-dimensional coordinates of the camera device system that can determine the positions within the image captured by the camera device; converting the three-dimensional coordinates of the camera device system into two-dimensional coordinates of the display system that can determine the positions within the image; and allocating the pixels constituting the image to the interpolated positions within the image determined by an interpolation method based on the two-dimensional coordinates of the camera device system and the two-dimensional coordinates of the display system.

[0023] The 12th aspect of the present invention is a program for causing a computer to perform a process comprising the following steps: given that the positional relationship between a three-dimensional ranging sensor and a camera device with a higher sampling resolution than the three-dimensional ranging sensor is known, obtaining, based on the ranging results of the three-dimensional ranging sensor for multiple measuring points, three-dimensional coordinates of the three-dimensional ranging sensor system that can determine the positions of the multiple measuring points, and defined by a three-dimensional coordinate system suitable for the three-dimensional ranging sensor; obtaining, based on the three-dimensional coordinates of the three-dimensional ranging sensor system, three-dimensional coordinates of the camera device system defined by a three-dimensional coordinate system suitable for the camera device; converting the three-dimensional coordinates of the camera device system into two-dimensional coordinates of the camera device system that can determine the positions within a photographic image obtained by the camera device; converting the three-dimensional coordinates of the camera device system into two-dimensional coordinates of the display system that can determine the positions within the frame; and allocating the pixels constituting the photographic image to the frame at interpolated positions determined by an interpolation method using the two-dimensional coordinates of the camera device system and the two-dimensional coordinates of the display system. Attached Figure Description

[0024] Figure 1 This is a schematic structural diagram illustrating an example of the overall structure of a moving body system.

[0025] Figure 2 This is a schematic perspective view showing an example of the detection axis of an accelerometer and an angular velocity sensor.

[0026] Figure 3 This is a block diagram illustrating an example of the hardware structure of an information processing system.

[0027] Figure 4 This is a block diagram illustrating one example of the main functions of a processor.

[0028] Figure 5 This is a conceptual diagram representing an example of the processing content of the acquisition department.

[0029] Figure 6 This is a conceptual diagram illustrating an example of how the LiDAR coordinate system is converted into a camera's 3D coordinate system.

[0030] Figure 7 This is a conceptual diagram representing an example of the processing content of the acquisition department.

[0031] Figure 8 This is a conceptual diagram representing an example of the processing content of the conversion unit.

[0032] Figure 9 This is a conceptual diagram illustrating how the camera's three-dimensional coordinates are projected onto the xy and uv planes using perspective.

[0033] Figure 10 This is a conceptual diagram representing an example of the processing content of the pixel allocation unit.

[0034] Figure 11 This is a flowchart illustrating an example of a texture mapping process.

[0035] Figure 12 This is a comparison diagram that compares a method that allocates only pixels corresponding to the TIN intersection point to the screen with a method that allocates pixels corresponding to the TIN intersection point and pixels corresponding to positions other than the TIN intersection point to the screen.

[0036] Figure 13 This is a conceptual diagram illustrating an example of how pixels, either the centroid or the centroid portion of the camera's two-dimensional coordinate system, are allocated to the image's two-dimensional coordinate system.

[0037] Figure 14 This is a conceptual diagram illustrating an example of how pixels on the edges of the camera's two-dimensional coordinate system are assigned to the image's two-dimensional coordinate system.

[0038] Figure 15 This is a block diagram illustrating one example of the main functionalities of the processor involved in the first variation.

[0039] Figure 16 This is a flowchart illustrating an example of the texture mapping process involved in the first variation.

[0040] Figure 17 This is a concept diagram illustrating an example of how a camera, moving from one position to another, can capture the same subject while simultaneously changing its position.

[0041] Figure 18 This is a conceptual diagram representing an example of the processing content of the processor involved in the second variation.

[0042] Figure 19 It is a conceptual diagram used to illustrate polar geometry.

[0043] Figure 20 This is a conceptual diagram illustrating an example of how texture mapping processing is mounted from a storage medium onto a computer. Detailed Implementation

[0044] Hereinafter, an example of an embodiment of the image processing apparatus, image processing method and program related to the present invention will be described with reference to the accompanying drawings.

[0045] First, let me explain the words and phrases used in the following description.

[0046] CPU stands for Central Processing Unit. GPU stands for Graphics Processing Unit. RAM stands for Random Access Memory. IC stands for Integrated Circuit. ASIC stands for Application Specific Integrated Circuit. PLD stands for Programmable Logic Device. FPGA stands for Field-Programmable Gate Array. SoC stands for System-on-a-chip. SSD stands for Solid State Drive. USB stands for Universal Serial Bus. HDD stands for Hard Disk Drive. EL stands for Electro-Luminescence. I / F stands for Interface. UI stands for User Interface. CMOS stands for Complementary Metal Oxide Semiconductor. CCD stands for Charge Coupled Device. LiDAR stands for Light Detection and Ranging. TIN stands for Triangulated Irregular Network. In this specification, an intersection point refers to the point where two adjacent edges of a polygon intersect (i.e., a vertex). Furthermore, in this specification, the location of a polygon or polygonal image block other than an intersection point refers to the location inside the polygon or polygonal image block. The inside of the polygon (within the polygon) also includes the edges of the polygon.

[0047] As an example, such as Figure 1 As shown, the information processing system 2 includes a mobile body 10 and an information processing device 20. A sensor unit 30 is mounted on the mobile body 10. An example of the mobile body 10 is an unmanned mobile body. Figure 1 In the example shown, an unmanned aerial vehicle (e.g., a drone) is shown as one example of a mobile body 10.

[0048] The mobile body 10 is used to measure and / or inspect land and / or infrastructure, etc. Examples of infrastructure include, for example, road facilities (e.g., bridges, road surfaces, tunnels, guardrails, traffic lights and / or windbreaks), waterway facilities, airport facilities, port facilities, water storage facilities, gas facilities, power supply facilities, medical facilities and / or fire protection facilities, etc.

[0049] Here, an unmanned aerial vehicle is cited as an example of the mobile body 10, but the technology of the present invention is not limited thereto. For example, the mobile body 10 can be a vehicle. Examples of vehicles include cable cars, aerial work platforms, or bridge inspection vehicles. Furthermore, the mobile body 10 can also be a slider or trolley capable of carrying the sensor unit 30. Moreover, the mobile body 10 can be a person. Here, a person refers, for example, to an operator who carries and operates the sensor unit 30 to measure and / or inspect land and / or infrastructure.

[0050] The information processing device 20 is a laptop computer. A laptop computer is shown here as an example, but it is only one example; it could also be a desktop computer. Furthermore, it is not limited to a personal computer; it could also be a server. The server could be a mainframe computer used locally with the mobile device 10, or it could be an external server implemented through cloud computing. Furthermore, the server could also be an external server implemented through network computing such as fog computing, edge computing, or grid computing.

[0051] The information processing device 20 includes a receiving device 22 and a display 24. The receiving device 22 includes a keyboard, mouse, and touch screen, and receives instructions from the user. The display 24 displays various information (e.g., images and characters). As an example of the display 24, an EL display (e.g., an organic EL display or an inorganic EL display) can be cited. However, it is not limited to EL displays and may also be other types of displays such as liquid crystal displays.

[0052] The mobile body 10 is connected to the information processing device 20 in a wireless communication manner, and various information is exchanged between the mobile body 10 and the information processing device 20 wirelessly.

[0053] The mobile body 10 has a main body 12 and multiple propellers 14. Figure 1 In the example shown, there are four propellers. The moving body 10 flies or hovers in three-dimensional space by controlling the rotation of each of the multiple propellers 14.

[0054] A sensor unit 30 is installed on the main body 12. Figure 1In the example shown, the sensor unit 30 is mounted on the upper part of the body 12. However, this is only one example, and the sensor unit 30 can also be mounted on a part other than the upper part of the body 12 (e.g., the lower part of the body 12).

[0055] The sensor unit 30 includes an external sensor 32 and an internal sensor 34. The external sensor 32 senses the external environment of the moving body 10. The external sensor 32 includes a LiDAR 32A and a camera 32B. The LiDAR 32A is an example of a "three-dimensional ranging sensor" according to the technology of this invention, and the camera 32B is an example of a "camera device" according to the technology of this invention.

[0056] The LiDAR32A scans the surrounding space by emitting pulsed laser beams L. The laser beam L can be, for example, visible or infrared light. The LiDAR32A receives reflected light from objects (e.g., natural and / or man-made objects) in the surrounding space, measures the time from the emission of the laser beam L to the reception of the reflected light, and calculates the distance to a measurement point on the object. Here, the measurement point is the point on the object where the laser beam L is reflected. Furthermore, each time the LiDAR32A scans the surrounding space, it outputs point cloud data representing multiple three-dimensional coordinates as positional information capable of determining the positions of multiple measurement points. The point cloud data is also referred to as a point cloud. The point cloud data is, for example, data represented by three-dimensional orthogonal coordinates.

[0057] The LiDAR32A, using the direction of travel of the moving body 10 as a reference, emits a laser beam L into a field of view S1 extending 135 degrees to the left and right and 15 degrees up and down. For example, the LiDAR32A emits the laser beam L into the entire field of view S1 while changing its angle by 0.25 degrees in either the left / right or up / down direction. The LiDAR32A repeatedly scans the field of view S1 and outputs the point cloud data for each scan. Hereinafter, for ease of explanation, the point cloud data output by the LiDAR32A during each scan will be referred to as the distinguished point cloud data PG (see reference). Figure 3 ).

[0058] Camera 32B is a digital camera equipped with an image sensor. Here, the image sensor refers to, for example, a CMOS image sensor or a CCD image sensor. Camera 32B captures a field of view S2 according to instructions given from an external source (e.g., information processing device 20). Capture based on camera 32B may include, for example, capturing moving images (e.g., capturing at a predetermined frame rate such as 30 frames per second or 60 frames per second) and capturing still images. Capture of moving images and capture of still images based on camera 32B are selectively performed according to instructions given to camera 32B from an external source.

[0059] The sampling resolution based on camera 32B (e.g., the density of pixels per unit area in a photographic image obtained by camera 32B) is higher than the sampling resolution based on LiDAR 32A (e.g., the density of ranging points per unit area in point cloud data). Furthermore, the positional relationship between LiDAR 32A and camera 32B is known in the information processing apparatus 20.

[0060] Furthermore, for ease of explanation, the following description assumes that the LiDAR 32A and camera 32B are fixed to the moving body 10. Also, for ease of explanation, the following description assumes that the field of view S2 is contained within the field of view S1 (i.e., the field of view S2 is a part of the field of view S1), the position of the field of view S2 relative to the field of view S1 is fixed, and the position of the field of view S2 relative to the field of view S1 is known. Furthermore, for ease of explanation, the example shown here is that the field of view S2 is contained within the field of view S1, but this is only one example; for example, the field of view S1 could be contained within the field of view S2 (i.e., the field of view S1 could be a part of the field of view S2). Specifically, the field of view S1 and the field of view S2 are configured to enable the texture mapping processing described later (see reference). Figure 11 The degree of overlap between the multiple distinguishing point cloud data (PG) and the camera image is sufficient.

[0061] The internal sensor 34 includes an accelerometer 36 and an angular velocity sensor 38. The internal sensor 34 detects physical quantities necessary to determine the direction, distance, and posture of the moving body 10, and outputs detection data representing the detected physical quantities. The detection data based on the internal sensor 34 (hereinafter also referred to as "detection data") includes, for example, acceleration data representing acceleration detected by the accelerometer 36 and acceleration data representing angular velocity detected by the angular velocity sensor 38.

[0062] The moving body 10 has a LiDAR coordinate system and a camera three-dimensional coordinate system. The LiDAR coordinate system is a three-dimensional coordinate system applicable to LiDAR32A (here, as an example, an orthogonal coordinate system in three-dimensional space). The camera three-dimensional coordinate system is a three-dimensional coordinate system applicable to camera 32B (here, as an example, an orthogonal coordinate system in three-dimensional space).

[0063] As an example, such as Figure 2 As shown, the orthogonal coordinate system applicable to the three-dimensional space of LiDAR32A, namely the LiDAR coordinate system, is composed of mutually orthogonal X coordinates. L Axis, Y L Axis and Z L The axes are defined. Point cloud data is defined by the LiDAR coordinate system.

[0064] Accelerometer 36 (reference) Figure 1) Detection applied to X L Axis, Y L Axis and Z L Acceleration in all directions of the axis. Angular velocity sensor 38 (reference) Figure 1 ) Detection around X L Axis, Y L Axis and Z L The angular velocity applied to each axis of the axis (i.e., each direction of roll, pitch, and yaw). That is, the internal sensor 34 is a six-axis inertial measurement sensor.

[0065] As an example, such as Figure 3 As shown, a controller 16, a communication I / F 18, and motors 14A are provided on the main body 12 of the mobile body 10. The controller 16 is implemented, for example, by an IC chip. Multiple motors 14A are provided on the main body 12. The multiple motors 14A are connected to multiple propellers 14. The controller 16 controls the flight of the mobile body 10 by controlling the multiple motors 14A. Furthermore, the controller 16 controls the scanning action of the LiDAR32A-based laser beam L.

[0066] External sensor 32 senses object 50, which is an example of the external environment. As a result, it outputs distinguishable point cloud data PG obtained by scanning the field of view S1 with a laser beam L by LiDAR 32A and a photographic image PD obtained by camera 32B, which represents the subject light in the field of view S2 (i.e., the reflected light representing the portion of object 50 contained within the field of view S2). Controller 16 receives distinguishable point cloud data PG and photographic image PD from external sensor 32.

[0067] The internal sensor 34 outputs detection data obtained through sensing (e.g., acceleration data based on the accelerometer 36 and angular velocity data based on the angular velocity sensor 38) to the controller 16. The controller 16 receives the detection data from the internal sensor 34.

[0068] The controller 16 wirelessly transmits the received point cloud data PG, camera image PD and detection data to the information processing device 20 via the communication I / F 18.

[0069] In addition to a receiving device 22 and a display 24, the information processing apparatus 20 also includes a computer 39 and a communication I / F 46. The computer 39 includes a processor 40, a storage device 42, and RAM 44. The receiving device 22, display 24, processor 40, storage device 42, RAM 44, and communication I / F 46 are connected to a bus 48. Furthermore, the information processing apparatus 20 is an example of an "image processing apparatus" according to the technology of this invention. The computer 39 is an example of a "computer" according to the technology of this invention. The processor 40 is an example of a "processor" according to the technology of this invention. The storage device 42 and RAM 44 are examples of "memory" according to the technology of this invention.

[0070] For example, processor 40 has a CPU and a GPU, and controls the entire information processing device 20. The GPU operates under the control of the CPU, and is responsible for performing tasks such as screen display and / or image processing. Alternatively, processor 40 can be one or more CPUs with integrated GPU functionality, or it can be one or more CPUs without integrated GPU functionality.

[0071] Storage device 42 is a non-volatile storage device for storing various programs and parameters. Examples of storage devices 42 include HDDs and SSDs. In addition, HDDs and SSDs are just examples; flash memory, magnetoresistive memory, and / or ferroelectric memory may also be used in conjunction with HDDs and / or SSDs.

[0072] RAM44 is a memory that temporarily stores information, and the processor 40 uses it as working memory. Examples of RAM44 include DRAM and / or SRAM.

[0073] Communication I / F46 wirelessly communicates with the mobile body 10 via Communication I / F18 to receive multiple distinguishable point cloud data PGs from the mobile body 10. The multiple distinguishable point cloud data PGs received via Communication I / F46 refer to multiple distinguishable point cloud data PGs acquired by the LiDAR32A at different times (i.e., multiple distinguishable point cloud data PGs obtained through multiple scans). Furthermore, Communication I / F46 wirelessly communicates with the mobile body 10 via Communication I / F18, and at each time the multiple distinguishable point cloud data PGs are acquired, receives the camera image PD and detection data obtained by the internal sensor 34. The distinguishable point cloud data PGs, camera image PD, and detection data received via Communication I / F46 are thus acquired and processed by the processor 40.

[0074] The processor 40 obtains synthetic point cloud data SG based on multiple distinguishable point cloud data PGs received from the mobile body 10. Specifically, the processor 40 performs a synthesis process to synthesize the multiple distinguishable point cloud data PGs received from the mobile body 10, thereby generating synthetic point cloud data SG. The synthetic point cloud data SG is a collection of multiple distinguishable point cloud data PGs obtained by scanning the field of view S1, and is stored by the processor 40 in the storage device 42. Furthermore, the multiple distinguishable point cloud data PGs are an example of the "distance measurement results of a three-dimensional ranging sensor for multiple measurement points" involved in the technology of this invention. And, the synthetic point cloud data SG is an example of the "three-dimensional coordinates of a three-dimensional ranging sensor system" involved in the technology of this invention.

[0075] As an example, such as Figure 4 As shown, a texture mapping processing program 52 is stored in storage device 42. Texture mapping processing program 52 is an example of a "program" according to the technology of this invention. Processor 40 reads texture mapping processing program 52 from storage device 42 and executes the read texture mapping processing program 52 on RAM 44. Processor 40 performs texture mapping processing according to the texture mapping processing program 52 executed on RAM 44 (see reference). Figure 11 The processor 40 operates as an acquisition unit 40A, a conversion unit 40B, and a pixel allocation unit 40C by executing the texture mapping processing program 52.

[0076] As an example, such as Figure 5 As shown, the acquisition unit 40A acquires the synthesized point cloud data SG from the storage device 42. Then, the acquisition unit 40A acquires the camera's three-dimensional coordinates defined by the camera's three-dimensional coordinate system based on the synthesized point cloud data SG. Here, the acquisition of the camera's three-dimensional coordinates based on the acquisition unit 40A is achieved by the acquisition unit 40A converting the synthesized point cloud data SG into camera three-dimensional coordinates. Here, the camera's three-dimensional coordinates are an example of the "three-dimensional coordinates of a camera device system" involved in the technology of this invention.

[0077] As an example, such as Figure 6 As shown, the transformation from synthetic point cloud data (SG) to camera 3D coordinates is achieved by converting the LiDAR coordinate system to the camera 3D coordinate system using a rotation matrix and a translation vector. Figure 6 The example shown illustrates how data determining the location of a measurement point P in synthetic point cloud data SG—i.e., three-dimensional coordinates (hereinafter also referred to as "LiDAR coordinates")—is converted into camera three-dimensional coordinates.

[0078] Here, the three axes of the LiDAR coordinate system are set as X... L Axis, Y L Axis and Z L The axis, the rotation matrix required to transform from the LiDAR coordinate system to the camera's 3D coordinate system is set toC L In the case of R, the rotation matrix C L R is represented by the following matrix (1), the rotation matrix C L R indicates that the moving body 10, together with the LiDAR coordinate system, revolves sequentially around the X-axis of the LiDAR coordinate system. L The axis, the Y-axis around the LiDAR coordinate system L The axis and the Z-axis around the LiDAR coordinate system L The position transformation of the moving body 10 is performed when the axis rotates by angles φ, θ, and ψ, respectively. Additionally, angles φ, θ, and ψ are calculated based on the angular velocity data included in the detection data.

[0079] [Formula 1]

[0080]

[0081] Then, set the origin of the LiDAR coordinate system to O. L Set the origin of the camera's 3D coordinate system to O. C Set the three axes of the camera's three-dimensional coordinate system to X. C Axis, Y C Axis and Z C The axis is defined as the three-dimensional coordinates (i.e., LiDAR three-dimensional coordinates) of the measurement point P in the LiDAR coordinate system. L Let P be the three-dimensional coordinates of the measurement point P in the camera's three-dimensional coordinate system (i.e., the camera's three-dimensional coordinates). C Let P be the translation vector required to transform the LiDAR coordinate system to the camera's 3D coordinate system. C L T, and the origin O L The position is represented in the LiDAR coordinate system as L O C In this case, the camera's three-dimensional coordinates C P is represented by the following equation (2). Additionally, the translation vector is calculated based on the acceleration data contained in the detection data. C L T.

[0082] [Formula 2]

[0083]

[0084] Here, C L T, C L R and L O CThe relationship is represented by the following equation (3). Therefore, by substituting equation (3) into equation (2), the camera's three-dimensional coordinates are obtained. C P is represented by the following equation (4). The synthetic point cloud data SG, which contains the LiDAR coordinates associated with the measurement point P, is converted into three-dimensional coordinates of multiple cameras using equation (4).

[0085] [Formula 3]

[0086]

[0087] [Formula 4]

[0088]

[0089] As an example, such as Figure 7 As shown, the acquisition unit 40A generates TIN54 based on the multiple camera 3D coordinates obtained by converting and synthesizing point cloud data SG using formula (4) into a digital data structure. TIN54 is a digital data structure representing a set of triangular image blocks 54A defined by a triangular mesh (e.g., an irregular triangular mesh). The multiple camera 3D coordinates define the positions of the intersections of the multiple triangular image blocks 54A contained in TIN54 (in other words, the vertices of each triangular image block 54A). Furthermore, the triangular image block 54A is an example of a "polygonal image block" according to the technology of this invention. Here, a triangular image block 54A is shown, but this is only one example, and it can also be a polygonal image block other than a triangle. That is, a polygonal set image block with a face structure formed by a set of multiple polygonal image blocks can be used instead of TIN54.

[0090] As an example, such as Figure 8 As shown, the conversion unit 40B converts the multiple camera three-dimensional coordinates used in TIN54 into multiple two-dimensional coordinates (hereinafter also referred to as "camera two-dimensional coordinates") capable of determining the position within the captured image PD. Furthermore, the conversion unit 40B converts the multiple camera three-dimensional coordinates used in TIN54 into multiple two-dimensional coordinates (hereinafter also referred to as "screen two-dimensional coordinates") capable of determining the position within the screen of the display 24. Here, the camera two-dimensional coordinates are an example of "camera device system two-dimensional coordinates" according to the technology of this invention, and the screen two-dimensional coordinates are an example of "display system two-dimensional coordinates" according to the technology of this invention.

[0091] As an example, such as Figure 9 As shown, the positions of each pixel constituting the camera image PD are determined by the camera 32B (reference). Figure 1 and Figure 3 The image sensor's imaging plane corresponds to the xy plane PDO (in) Figure 9 In the example shown, it is equivalent to starting from the origin O C In ZC The coordinates within the plane (the image plane that is only separated by the focal length along the axial direction) are determined. The xy plane PDO is a plane that can be represented by a two-dimensional coordinate system (hereinafter also referred to as the "camera two-dimensional coordinate system") defined by the x-axis and y-axis. The position of each pixel constituting the image PD is determined by the camera two-dimensional coordinates (x, y). Here, with i = 0, 1, 2, ..., n, the position A of the pixel in the xy plane PD0 is... i A i+1 and A i+2 The positions P of the three vertices of the triangular image block 54A i P i+1 and P i+2 Corresponding. Position A i A i+1 and A i+2 The camera's two-dimensional coordinates (x0, y0), (x1, y1), and (x2, y2) are obtained through position P. i P i+1 and P i+2 The three-dimensional coordinates of each camera are obtained by perspective projection onto the xy plane PD0.

[0092] exist Figure 9 The example shown illustrates setting the origin to O. d And have mutually orthogonal X d Axis, Y d Axis and Z d The display uses a 3D coordinate system. The display's 3D coordinate system is applicable to a 24-inch display (reference). Figure 1 and Figure 3 The three-dimensional coordinates of the object 50 are defined. A screen 24A is set in the three-dimensional coordinate system of the display. Screen 24A is the screen of the display 24, displaying an image representing the object 50. For example, the image displayed on screen 24A is a texture image. Here, a texture image refers to an image obtained by capturing a photographic image PD using a camera 32B, which is then texture-mapped onto screen 24A.

[0093] The position and orientation of the screen 24A set in the three-dimensional coordinate system of the display are, for example, determined by the receiving device 22 (reference). Figure 3 The position of each pixel constituting the image 24A is changed according to the received instruction. The position of each pixel is determined by the uv plane 24A1 (in...). Figure 9 In the example shown, from the origin O d In Z d The coordinates of the pixels in the display 24A1 are determined by a two-dimensional coordinate system (hereinafter also referred to as the "screen two-dimensional coordinate system") defined by the u-axis and v-axis. The positions of each pixel constituting the screen 24A1, i.e., its position within the screen of the display 24, are determined by the screen two-dimensional coordinates (u, v).

[0094] exist Figure 10 In the example shown, the position of the pixel within the uv plane 24A1 is B. i B i+1 and B i+2 The positions P of the three vertices of the triangular image block 54A i P i+1 and P i+2 Corresponding. Position B i B i+1 and B i+2 The two-dimensional coordinates (u0, v0), (u1, v1), and (u2, v2) of the image pass through position P. i P i+1 and P i+2 The three-dimensional coordinates of each camera are obtained by perspective projection onto the uv plane 24A1.

[0095] As an example, such as Figure 10 As shown, the pixel allocation unit 40C allocates pixels from the multiple pixels contained in the captured image PD (all pixels, for example) to the multiple triangular image blocks 54A contained in TIN54 (reference). Figures 7-9 The pixels corresponding to the intersection points of the images 24A and 54A are assigned to the intersection points within the image, thereby creating a 3D image (i.e., a texture image perceived by the user in 3D through image 24A). The intersection point within the image refers to the position of one of the multiple pixels (in one example, all pixels) contained in image 24A that intersects with the multiple triangular image blocks 54A contained in TIN54 (see reference 54A). Figures 7-9 The location corresponding to the intersection point of ) . In Figure 10 In the example shown, position A is one of the multiple pixels contained in the camera image PD. i A i+1 and A i+2 (That is, each pixel, whose position is determined by the camera's two-dimensional coordinates (x0, y0), (x1, y1), and (x2, y2),) is assigned to position B within the image 24A. i B i+1 and B i+2 (That is, the position determined by the two-dimensional coordinates (u0, v0), (u1, v1), and (u2, v2) on the screen).

[0096] Thus, even if the pixels in the multiple pixels contained in the camera image PD that correspond to the positions of the intersection points of the multiple triangular image blocks 54A contained in TIN54 are allocated to the corresponding positions within the screen 24A, the pixel density of the image displayed on the screen 24A is correspondingly lower because no pixels are allocated to positions other than the intersection points of the multiple triangular image blocks 54A (in other words, the vertices of the triangular image blocks 54A).

[0097] Therefore, the pixel allocation unit 40C allocates the pixels constituting the captured image PD to the interpolation positions within the frame 24A, determined by an interpolation method using camera two-dimensional coordinates and frame two-dimensional coordinates. Here, the interpolation position refers to the position corresponding to any location other than the intersection of the triangular image block 54A (in...). Figure 10 In the example shown, the second interpolation position is D0. Examples of interpolation methods include linear interpolation. Other interpolation methods besides linear interpolation include polynomial interpolation and spline interpolation.

[0098] exist Figure 10 In the example shown, among the positions of multiple pixels (all pixels in one example) contained in the camera image PD, the first interpolation position C0 is shown as the position corresponding to the position other than the intersection of the triangular image block 54A. Furthermore, in Figure 10 In the example shown, among the multiple pixels (all pixels as an example) contained in frame 24A, the second interpolation position D0 is shown as the position corresponding to all positions except the intersection of the triangular image block 54A. The second interpolation position D0 corresponds to the first interpolation position C0. Furthermore, in Figure 10 In the example shown, the camera's two-dimensional coordinates (x, y) are shown as the first interpolation position C0, and the image's two-dimensional coordinates (u, v) are shown as the second interpolation position D0.

[0099] The first interpolation position C0 is located within the camera image PD at position A. i A i+1 and A i+2 This refers to the position inside the triangle formed by the three vertices (i.e., inside the triangular image block formed within the camera image PD). For example, the first interpolation position C0 is the position between position C1 and position C2. Position C1 is the position within the camera image PD that is located at position A. i A i+1 and A i+2 Connect position A among the three sides of a triangle with three vertices. i and A i+2 The position is located on the edge of the image. Position C2 exists within the camera image PD at position A. i A i+1 and A i+2 Connect position A among the three sides of a triangle with three vertices. i and A i+1 The positions on the side. Additionally, positions C1 and C2 can be determined by the receiving device 22 (reference). Figure 3 The received instruction can be used to determine this, or it can be based on the applicable connection location A. i and A i+2 The edge and the connection position A i and A i+1The ratio of the inner sides is determined by the specified proportions.

[0100] The two-dimensional coordinates of the camera at position C1 are (x 02 y 02 The two-dimensional coordinates of the camera at position C2 are (x... 01 y 01 The first interpolation position C0 is determined using the camera's two-dimensional coordinates (x0, y0), (x1, y1), (x2, y2), (x...). 01 y 01 ) and (x 02 y 02 The two-dimensional coordinates (x, y) of the camera were determined by interpolation.

[0101] The second interpolation position D0 exists within image 24A at position B. i B i+1 and B i+2 This refers to the position inside the triangle formed by the three vertices (i.e., inside the triangular image block formed within frame 24A). For example, the second interpolation position D0 is the position between position D1 and position D2. Position D1 is the position within frame 24A that exists at position B. i B i+1 and B i+2 Connect position B among the three sides of a triangle with three vertices. i and B i+2 The position D2 exists within the frame 24A at position B. i B i+1 and B i+2 Connect position B among the three sides of a triangle with three vertices. i and B i+1 On the edge. Additionally, position D1 corresponds to position C1, and position D2 corresponds to position C2.

[0102] The two-dimensional coordinates of position D1 on the screen are (u 02 v 02 The two-dimensional coordinates of the camera at position C2 are (u 01 v 01 The second interpolation position D0 is based on the two-dimensional coordinates (u0, v0), (u1, v1), (u2, v2), and (u...). 01 v 01 ) and (u 02 v 02 The two-dimensional coordinates (u, v) of the image were determined by interpolation.

[0103] In a video image PD containing multiple pixels, the pixel at the first interpolation position C0 determined by the camera's two-dimensional coordinates (x, y) is assigned by the pixel allocation unit 40C to the second interpolation position D0 determined by the image's two-dimensional coordinates (u, v). The pixel at the first interpolation position C0 can be one of the pixels constituting the video image PD itself, but if no pixel exists at the position determined by the camera's two-dimensional coordinates (x, y), multiple pixels adjacent to the position determined by the camera's two-dimensional coordinates (x, y) (e.g., at position A) will be assigned. i A i+1 and A i+2 The pixel generated by interpolating the pixels of the three vertices can be used as the pixel of the first interpolation position X0.

[0104] Furthermore, it is preferable that the resolution of the photographic image PD (i.e., the sampling resolution based on camera 32B) is higher than the resolution of the synthesized point cloud data SG (i.e., the sampling resolution based on LiDAR 32A), such that there are more than a number of pixels within the triangles (i.e., the triangles corresponding to the triangle image block 54A) within the photographic image PD, as determined by the resolution of the interpolation processing (e.g., the process of determining the first interpolation position C0 using interpolation). Here, the number determined by the resolution of the interpolation processing refers, for example, a number predetermined in advance such that the higher the resolution of the interpolation processing, the more pixels there are. The resolution of the interpolation processing can be a fixed value determined based on the resolution of the captured image PD and / or the screen 24A, or a variable value that changes according to an external instruction (e.g., an instruction received by the receiving device 22). It can also be a variable value that changes according to the size (e.g., average size) of the triangle image block 54A, the triangle obtained by perspective projection of the triangle image block 54A onto the camera's two-dimensional coordinate system, and / or the triangle obtained by perspective projection of the triangle image block 54A onto the screen's two-dimensional coordinate system. When the resolution of the interpolation processing depends on the triangle image block 54A, for example, a larger triangle image block 54A results in a higher resolution for the interpolation processing, and a smaller triangle image block 54A results in a lower resolution for the interpolation processing.

[0105] In order to allocate the pixel at the first interpolation position C0 (i.e., the pixel at the first interpolation position C0 determined according to the camera's two-dimensional coordinates (x, y)) to the second interpolation position D0 (i.e., the second interpolation position D0 determined according to the image's two-dimensional coordinates (u, v)), the pixel allocation unit 40C calculates the image's two-dimensional coordinates (u, v) at the second interpolation position D0 using the following formulas (5) to (12).

[0106] [Formula 5]

[0107]

[0108] [Formula 6]

[0109]

[0110] [Formula 7]

[0111]

[0112] [Formula 8]

[0113]

[0114] [Formula 9]

[0115]

[0116] [Formula 10]

[0117]

[0118] [Formula 11]

[0119]

[0120] [Formula 12]

[0121]

[0122] Next, as for the function of information processing system 2, refer to Figure 11 The texture mapping process performed by the processor 40 of the information processing device 20 will be described.

[0123] exist Figure 11 An example of a texture mapping process performed by processor 40 is shown in the figure. Figure 11 The texture mapping process shown is an example of the "image processing method" involved in the technology of this invention. Furthermore, the following description assumes that the synthetic point cloud data SG is already stored in the storage device 42.

[0124] exist Figure 11 In the texture mapping process shown, firstly, in step ST100, the acquisition unit 40A acquires the composite point cloud data SG (reference) from the storage device 42. Figure 5 After performing the processing in step ST100, the texture mapping process is transferred to step ST102.

[0125] In step ST102, the acquisition unit 40A converts the synthesized point cloud data SG into three-dimensional coordinates of multiple cameras (reference). Figure 5 and Figure 6 After performing the processing in step ST102, the texture mapping process moves on to step ST104.

[0126] In step ST104, the acquisition unit 40A generates TIN54 (reference) based on the three-dimensional coordinates of multiple cameras. Figure 7 After performing the processing in step ST104, the texture mapping process moves on to step ST106.

[0127] In step STl06, the conversion unit 40B converts the multiple camera 3D coordinates used in TIN54 into multiple camera 2D coordinates (see reference). Figure 8 and Figure 9 After performing the processing in step ST106, the texture mapping process moves on to step ST108.

[0128] In step ST108, the conversion unit 40B converts the multiple camera 3D coordinates used in TIN54 into multiple image 2D coordinates (see reference). Figure 8 and Figure 9 After performing the processing in step ST108, the texture mapping process is transferred to step ST110.

[0129] In step ST110, the pixel allocation unit 40C assigns a plurality of pixels contained in the camera image PD to a plurality of triangular image blocks 54A contained in TIN54 (reference). Figures 7-9 The pixels corresponding to the intersection points of the graph are assigned to the intersection points within the screen, thereby creating a texture image. The texture image obtained by performing step ST110 is displayed on screen 24A.

[0130] Next, in order to refine the texture image obtained by performing the processing in step ST110, the pixel allocation unit 40C performs the processing in step ST112 and step ST114.

[0131] In step ST112, the pixel allocation unit 40C calculates the two-dimensional coordinates (u, v) of the interpolation position D0 within the image 24A, i.e., the second interpolation position D0, using the camera two-dimensional coordinates obtained in step ST106, the image two-dimensional coordinates obtained in step ST108, and the interpolation method of formulas (5) to (12). (See reference) Figure 10 After performing the processing in step ST112, the texture mapping process moves on to step ST114.

[0132] In step ST114, the pixel allocation unit 40C allocates the two-dimensional coordinates (u, v) of the interpolation position (u, v) within the image 24A, i.e., the second interpolation position D0, to the camera two-dimensional coordinates (in the image 24A) of the multiple pixels constituting the image PD, corresponding to the two-dimensional coordinates (u, v) of the image. Figure 10In the example shown, the camera's two-dimensional coordinates (x, y) are represented by pixels. Thus, a texture image with a finer texture than the texture image obtained by performing step ST110 is displayed on screen 24A. After performing step ST114, the texture mapping process ends.

[0133] As explained above, in information processing system 2, the synthesized point cloud data SG is converted into three-dimensional coordinates of multiple cameras (see reference). Figures 5-7 The three-dimensional coordinates of multiple cameras are converted into two-dimensional coordinates of multiple cameras and two-dimensional coordinates of multiple images (see reference). Figure 8 and Figure 9 Then, the pixels constituting the camera image PD are allocated to the frame 24A, and the interpolation positions of the multiple camera three-dimensional coordinates are determined using an interpolation method that utilizes multiple camera two-dimensional coordinates and multiple frame two-dimensional coordinates (in... Figure 10 In the example shown, the second interpolation position is D0 (reference). Figure 10 Therefore, a three-dimensional image (i.e., a texture image perceived by the user in three dimensions) is displayed within screen 24A. Thus, according to this structure, compared to creating an image within the screen solely using synthetic point cloud data SG, it is possible to achieve high-precision creation of the image within screen 24A. Furthermore, since screen 24A is the screen of display 24, the texture image can be displayed to the user via display 24.

[0134] Furthermore, in the information processing system 2, the multiple pixels contained in the camera image PD correspond to the multiple triangular image blocks 54A contained in TIN54 (reference). Figures 7-9 The pixels corresponding to the intersection points of the two-dimensional coordinates (u, v) are assigned to the intersection points within the frame, thus creating a texture image. Then, for the frame's two-dimensional coordinates (u, v) at the interpolation position D0 within frame 24A, the camera's two-dimensional coordinates (in the image) corresponding to the frame's two-dimensional coordinates (u, v) are assigned from among the pixels constituting the image PD. Figure 10 In the example shown, the pixels are at the camera's two-dimensional coordinates (x, y). Thus, as an example, ... Figure 12 As shown, if we compare an example of allocating pixels corresponding only to the intersection points of the multiple triangular image blocks 54A contained in TIN54 to the intersection points on the screen, and an example of allocating pixels corresponding to the intersection points of the multiple triangular image blocks 54A contained in TIN54 and pixels corresponding to positions other than the intersection points of the multiple triangular image blocks 54A contained in TIN54 to the screen 24A, then the latter example can obtain a texture image with a higher pixel density than the former example. That is, according to this structure, a texture image with a finer texture than one created by allocating pixels corresponding only to the intersection points of the multiple triangular image blocks 54A contained in TIN54 to the intersection points on the screen can be displayed on the screen 24A.

[0135] Furthermore, in information processing system 2, a TIN54 defined by multiple triangular image blocks 54A is generated based on the three-dimensional coordinates of multiple cameras. The positions of the intersection points of the multiple triangular image blocks 54A are defined by the three-dimensional coordinates of the multiple cameras. The positions of the intersection points of the multiple triangular image blocks 54A contained in TIN54 are perspective-projected onto the xy plane PD0 and the uv plane 24A1. Then, position A is obtained by perspective-projecting the positions of the intersection points of the multiple triangular image blocks 54A onto the xy plane PDO. i A i+1 and A i+2 The position B of each pixel is obtained by perspective projection onto the uv plane 24A1, where the positions of the intersections of multiple triangular image blocks 54A are assigned to each pixel. i B i+1 and B i+2 Therefore, a texture image is created within the frame 24A. Thus, according to this structure, texture images can be created more easily compared to creating texture images without utilizing polygonal image blocks such as triangular image blocks 54A.

[0136] Furthermore, in the information processing system 2, the second interpolation position D0 is applied to positions within the frame 24A that correspond to all positions except the intersection point of the triangular image block 54A. That is, the pixels constituting the camera image PD are assigned not only to the intersection point within the frame but also to the second interpolation position D0. Therefore, according to this structure, compared to the case where the pixels constituting the camera image PD are only assigned to the intersection point within the frame, a finer texture image can be displayed on the frame 24A.

[0137] Furthermore, in information processing system 2, the synthesized point cloud data SG is converted into multiple camera 3D coordinates, which are then converted into multiple camera 2D coordinates and multiple image 2D coordinates. Texture mapping is then achieved by assigning pixels at positions determined by the camera 2D coordinates to positions determined by the image 2D coordinates. Therefore, according to this structure, a higher precision texture image can be obtained compared to the case where texture mapping is performed without converting the synthesized point cloud data SG into multiple camera 3D coordinates.

[0138] Furthermore, in the above embodiment, an example of applying one second interpolation position D0 to each triangular image block 54A was described, but the technology of the present invention is not limited thereto. For example, multiple second interpolation positions D0 may be applied to each triangular image block 54A. In this case, multiple first interpolation positions C0 are given for each triangular image block 54A. Then, pixels corresponding to the multiple first interpolation positions C0 are assigned to the multiple second interpolation positions D0. Thus, compared to the case where only one second interpolation position D0 is applied to a triangular image block 54A, a more refined texture image can be created as the image displayed on the screen 24A.

[0139] Furthermore, as an example, such as Figure 13 As shown, in the camera's two-dimensional coordinate system, at least one second interpolation position D0 can be applied to the locations corresponding to the centroid CG1 and / or the centroid portion CG2 of the triangular image block 54A (e.g., a circular portion centered on the centroid CG1 in a region more inward than the triangular image block 54A). In this case, for example, in the camera's two-dimensional coordinate system, at least one first interpolation position C0 is given to the locations corresponding to the centroid CG1 and / or the centroid portion CG2 of each triangular image block 54A. Then, in the same manner as in the above embodiment, pixels corresponding to the first interpolation position C0 are assigned to at least one second interpolation position D0. Thus, in the camera's two-dimensional coordinate system, compared to the case where the second interpolation position D0 is applied to the location corresponding to the intersection point of the triangular image block 54A closer to the centroid CG1 of the triangular image block 54A, pixel deviation in the image displayed on the screen 24A can be suppressed. And, as an example, as Figure 14 As shown, at least one second interpolation position D0 can be applied to the portion corresponding to the edge 54A1 of the triangular image block 54A. In this case, for example, the portion corresponding to the edge 54A1 of the triangular image block 54A (in...) Figure 10 In the example shown, positions C1 and / or C2) give at least one first interpolation position C0. Then, in the same manner as in the above embodiment, pixels corresponding to the first interpolation position C0 are assigned to at least one second interpolation position D0.

[0140] Furthermore, the number of interpolation positions applicable to the set of triangles corresponding to TIN54 in both the camera 2D coordinate system and the screen 2D coordinate system (e.g., the average number of interpolation positions applicable to the triangular image block 54A) can be changed according to an external instruction (e.g., an instruction received by the receiving device 22). In this case, if further refinement of the image displayed on the screen 24A is desired, the number of interpolation positions applicable to the triangles corresponding to the triangular image block 54A in both the camera 2D coordinate system and the screen 2D coordinate system is increased; if the computational load in texture mapping processing is to be reduced, the number of interpolation positions is decreased.

[0141] [First Variation]

[0142] In the above embodiments, an example has been described in which the acquisition unit 40A, the conversion unit 40B, and the pixel allocation unit 40C operate by the processor 40 executing the texture mapping processing program 52. However, the technology of the present invention is not limited to this, for example, Figure 15 As shown, the processor 40 can also function as a control unit 40D by executing the texture mapping processing program 52.

[0143] As an example, such as Figure 15 As shown, the control unit 40D acquires synthetic point cloud data SG for conversion to multiple camera 3D coordinates via the acquisition unit 40A, and also acquires the multiple camera 3D coordinates obtained by the acquisition unit 40A. Based on the synthetic point cloud data SG and the multiple camera 3D coordinates, the control unit 40D generates a dataset 56, which is an example of "correspondence association information" according to the technology of this invention. The dataset 56 is information obtained by establishing a correspondence association between the synthetic point cloud data SG before conversion to multiple camera 3D coordinates by the acquisition unit 40A and the multiple camera 3D coordinates after conversion by the acquisition unit 40A.

[0144] The control unit 40D stores the dataset 56 in the storage device 42. Thus, the storage device 42 maintains the dataset 56.

[0145] As an example, such as Figure 16 As shown, the texture mapping process involved in the first variation is similar to... Figure 11 The difference between the texture mapping process shown is the inclusion of step ST200 and steps ST202 to ST206. Step ST200 is performed between steps ST102 and ST104. Steps ST202 to ST206 are performed after step ST114.

[0146] exist Figure 16In the texture mapping process shown, in step ST200, the control unit 40D generates a dataset 56 by establishing a corresponding association between the synthetic point cloud data SG obtained in step ST100 and the three-dimensional coordinates of multiple cameras obtained in step ST102, and stores the generated dataset 56 in the storage device 42.

[0147] In step ST202, the control unit 40D determines whether the synthetic point cloud data SG contained in the dataset 56 in the storage device 42 has been selected. The selection of the synthetic point cloud data SG is, for example, based on the receiving device 22 (reference). Figure 3 The received instruction is used to implement this. In step ST202, if the synthetic point cloud data SG contained in the dataset 56 within the storage device 42 is not selected, the determination is negative, and the texture mapping process proceeds to step ST206. In step ST202, if the synthetic point cloud data SG contained in the dataset 56 within the storage device 42 is selected, the determination is positive, and the texture mapping process proceeds to step ST204.

[0148] In step ST204, the acquisition unit 40A acquires multiple camera 3D coordinates that are associated with the synthetic point cloud data SG selected in step ST202 from the dataset 56 in the storage device 42, and generates a TIN 54 based on the acquired multiple camera 3D coordinates. After the processing in step ST204 is executed, the texture mapping processing is transferred to step ST106. In steps ST106 to ST110, processing using the TIN 54 generated in step ST204 is performed. Then, after the processing in step ST110 is executed, the processing in steps ST112 and ST114 is performed.

[0149] In step ST206, the control unit 40D determines whether the conditions for ending the texture mapping process (hereinafter referred to as the "end condition") are met. As a first example of the end condition, the receiving device 22 (see reference 206) can be used. Figure 3 The conditions include receiving an instruction to end the texture mapping process. As a second example of an end condition, a specified time (e.g., 15 minutes) may be elapsed after the determination in step ST202 has not been affirmed since the start of texture mapping process.

[0150] In step ST206, if the termination condition is not met, the determination is negative, and the texture mapping process proceeds to step ST202. In step ST206, if the termination condition is met, the determination is positive, and the texture mapping process ends.

[0151] As explained above, in the first variation, the storage device 42 maintains the dataset 56 as information obtained by establishing a corresponding association between the synthetic point cloud data SG and the three-dimensional coordinates of multiple cameras. Therefore, according to this structure, for the parts of the object 50 for which the synthetic point cloud data SG has already been obtained, texture mapping processing can be performed based on the three-dimensional coordinates of multiple cameras that are associated with the already obtained synthetic point cloud data SG without further scanning by LiDAR.

[0152] Furthermore, in the first variant, TIN54 (reference dataset 56) is generated by processor 40 from reference dataset 56. Figure 16 Step ST204) is executed, followed by the processing after step ST106. Thus, the pixels constituting the camera image PD are allocated to the frame 24A according to the interpolation position of the multiple camera three-dimensional coordinates determined by an interpolation method using multiple camera two-dimensional coordinates and multiple frame two-dimensional coordinates (in... Figure 10 In the example shown, the second interpolation position is D0. Therefore, according to this structure, for the parts of object 50 for which synthetic point cloud data SG has already been obtained, a three-dimensional image (i.e., a texture image perceived by the user in three dimensions) can be created and displayed on screen 24A without further scanning by LiDAR.

[0153] [Second Variation]

[0154] In the above embodiments, an example of converting synthetic point cloud data SG into three-dimensional coordinates of multiple cameras has been given. However, the technology of the present invention is not limited to this. Multiple three-dimensional coordinates of multiple cameras can also be calculated from multiple camera images PD.

[0155] In this case, firstly, as an example, such as Figure 17 As shown, the movable body 10 is moved to multiple positions, and the camera 32B captures images of the subject 58 from each position. For example, the movable body 10 is moved sequentially to the first camera position, the second camera position, and the third camera position, and the camera 32B captures images of the subject 58 from the first camera position, the second camera position, and the third camera position, respectively.

[0156] An image of the subject 58 is formed on the imaging surface 32B1, located only a focal length away from the center of the lens of camera 32B, and the image of the subject is captured by camera 32B. Figure 17 In the example shown, a subject image OI1 corresponding to the subject 58 is imaged on the camera surface 32B1 at the first camera position. Furthermore, a subject image OI2 corresponding to the subject 58 is imaged on the camera surface 32B1 at the second camera position. And, a subject image O13 corresponding to the subject 58 is imaged on the camera surface 32B1 at the third camera position.

[0157] The image PD obtained by the camera 32B at the first camera position contains an electronic image equivalent to the subject image OI1; the image PD obtained by the camera 32B at the second camera position contains an electronic image equivalent to the subject image OI2; and the image PD obtained by the camera 32B at the third camera position contains an electronic image equivalent to the subject image OI3.

[0158] As an example, such as Figure 18 As shown, the processor 40 acquires multi-frame video images PD from the camera 32B. Here, multi-frame video images PD refer to multiple video images PD obtained, for example, by the camera 32B capturing the subject 58 from at least two different positions, such as a first camera position, a second camera position, and a third camera position. The processor 40 calculates the camera's three-dimensional coordinates, which determine the positions of the feature points of the subject 58, based on the feature points of the subject 58 contained as images (here, as an example, electronic images) between the multi-frame video images PD, thereby acquiring the camera's three-dimensional coordinates. The multi-frame video images PD are obtained by the camera 32B capturing the subject 58 from different positions.

[0159] exist Figure 18 In the example shown, feature point Q contained in subject 58 is illustrated. The first image PD1, obtained by capturing subject 58 at the first camera position using camera 32B, contains feature point q1 within the first image, corresponding to feature point Q. Furthermore, the second image PD2, obtained by capturing subject 58 at the second camera position using camera 32B, contains feature point q2 within the second image, corresponding to feature point Q. The camera's three-dimensional coordinates (X, Y, Z) of feature point Q are calculated based on the two-dimensional coordinates (x3, y3) of feature point q1 within the camera's field of view in the first image and the two-dimensional coordinates (x4, y4) of feature point q3 within the camera's field of view in the second image. That is, the processor 40 calculates the position and pose of the camera 32B geometrically based on the two-dimensional coordinates (x3, y3) of feature point q1 in the first camera image and the two-dimensional coordinates (x4, y4) of feature point q2 in the second camera image, and calculates the three-dimensional coordinates (X, Y, Z) of feature point Q based on the calculation results.

[0160] As an example, such as Figure 19As shown, the position and orientation of camera 32B are determined by polar plane 60, which is a triangular plane with feature point Q, camera center O1, and camera center O2 as its three vertices. Camera center O1 exists on a straight line from feature point Q through feature point q1 in the first image, and camera center O2 exists on a straight line from feature point Q through feature point q2 in the second image. Polar plane 60 has a baseline BL. Baseline BL is a line segment connecting camera center O1 and camera center O2. Furthermore, polar plane 60 has poles e1 and e2. Pole e1 is the point where baseline BL intersects the plane of the first image PD1, and pole e2 is the point where baseline BL intersects the plane of the second image PD2. The first image PD1 has an epipolar line EP1. Epipolar line EP1 is a straight line passing through pole e1 and feature point q1 in the first image. The second image PD2 has an epipolar line EP2. The epipolar line EP2 is a straight line passing through the pole e2 and the feature point q2 in the second camera image.

[0161] Thus, when camera 32B captures images of a subject 58 existing in three-dimensional space from multiple camera positions (here, for example, the first camera position and the second camera position) (i.e., feature point Q is projected onto the plane of the first camera image PD1 as feature point q1 within the first camera image, and feature point Q is projected onto the plane of the second camera image PD2 as feature point q2 within the second camera image), polar geometry emerges between the first camera image PD1 and the second camera image PD2 as a unique geometry based on the polar plane 60. Polar geometry represents the correlation between feature point Q, feature point q1 within the first camera image, and feature point q2 within the second camera image. For example, if the position of feature point Q changes, feature points q1 and q2 within the first and second camera images change, and the polar plane 60 also changes. If the polar plane 60 changes, poles e1 and e2 also change, and therefore epipolar lines EP1 and EP2 also change. That is, the polar geometry contains the information needed to determine the position and orientation of the camera 32B at multiple camera positions (here, for example, the first camera position and the second camera position).

[0162] Therefore, the processor 40 uses the two-dimensional coordinates (x3, y3) of feature point q1 in the first camera image, the two-dimensional coordinates (x4, y4) of feature point q2 in the second camera image, and polar geometry to calculate the fundamental matrix E according to the following formulas (13) to (19), and derives the rotation matrix R and translation vector T based on the calculated fundamental matrix E. Then, the processor 40 calculates the camera three-dimensional coordinates (X, Y, Z) of feature point Q based on the derived rotation matrix R and translation vector T.

[0163] Here, an example of the method for calculating the basic matrix E is explained. The relationship between the camera three-dimensional coordinates (X1, Y1, Z1) of the feature point Q represented by the coordinate system of the camera 32B at the first camera position and the camera three-dimensional coordinates (X2, Y2, Z2) of the feature point Q represented by the coordinate system of the camera 32B at the second camera position is expressed by the following equation (13). In equation (13), Q1 is the camera three-dimensional coordinates (X1, Y1, Z1) of feature point Q represented by the coordinate system of camera 32B at the first camera position, Q2 is the camera three-dimensional coordinates (X2, Y2, Z2) of feature point Q represented by the coordinate system of camera 32B at the second camera position, R is the rotation matrix (i.e., the rotation matrix required to transform from the coordinate system of camera 32B at the first camera position to the coordinate system of camera 32B at the second camera position), and T is the translation vector (i.e., the translation vector required to transform from the coordinate system of camera 32B at the first camera position to the coordinate system of camera 32B at the second camera position).

[0164] [Formula 13]

[0165] Q2 = RQ1 + T……(13)

[0166] The projection of the feature point Q from the camera's three-dimensional coordinates (X1, Y1, Z1) to the two-dimensional coordinates (x3, y3) and (x4, y4) is represented by the following equation (14). In equation (14), λ1 represents the depth loss when the feature point Q is projected onto the first camera image PD1, and λ2 represents the depth loss when the feature point Q is projected onto the second camera image PD2.

[0167] [Formula 14]

[0168]

[0169] The expression (14) is replaced by the following expression (15). If the expression (15) is substituted into the expression (13), the following expression (16) can be obtained.

[0170] [Formula 15]

[0171] λ2q2=Q2, λ1=Q1 (λ1=Z1, λ2=Z2)……(15)

[0172] [Formula 16]

[0173] λ2q2=Rλ1+T……(16)

[0174] The translation vector T represents the distance and orientation from camera center O2 to camera center O1. That is, the translation vector T is equivalent to the vector extending from camera center O2 to camera center O1, which coincides with the baseline BL. On the other hand, "λ2q2" (i.e., "Q2" in equation (15)) on the left side of equation (16) is equivalent to the straight line extending from camera center O2 to feature point Q, which coincides with one side of the polar plane 60. Therefore, the cross product of the translation vector T and "λ2q2" becomes a vector W perpendicular to the polar plane 60 (see reference). Figure 19 Here, if the right side of the logarithmic equation (16) is also taken as the outer product with the translation vector T, it is represented by the following equation (17). In addition, in equation (17), the translation vector T is represented as a distorted symmetric matrix [T].

[0175] [Formula 17]

[0176] T×λ2q2=T×(Rλ1q1+T)

[0177] [T|×λ2q2=[T]×Rλ1q1……(17)

[0178] Vector W (reference) Figure 19 Since the polar plane 60 is orthogonal, if we take the inner product of both sides of the logarithmic equation (17) with the two-dimensional coordinates (x4, y4) of the feature point q2 in the second image, then as shown in the following equation (18), it becomes "0". λ1 and λ2 are constant terms and do not affect the calculation of the rotation matrix R and the translation vector T, so they are deleted from the equation (18). If we set it as "E=[T]×R", then we can derive the equation (19) representing the so-called polar constraint from the equation (18).

[0179] [Formula 18]

[0180]

[0181]

[0182] [Formula 19]

[0183]

[0184] The fundamental matrix E is calculated by simultaneously solving multiple extreme constraints derived for each of the multiple feature points, including feature point Q. Since the fundamental matrix E contains only the rotation matrix R and the translation vector T, the rotation matrix R and the translation vector T can also be obtained by calculating the fundamental matrix E. The rotation matrix R and the translation vector T are derived from the fundamental matrix E using known methods such as singular value decomposition (e.g., see https: / / ir.lib.hiroshima-u.ac.jp / 00027688, 20091125note_reconstruction.pdf_Image Engineering Lecture Notes: 3D Restoration Using Linear Algebra_Tetsu Tamaki_November 25, 2018_page 59 “1.4.2_Polar Geometry Based on Projective Geometry”~page 69 “1.4.4_Summary of 3D Restoration”).

[0185] The camera 3D coordinates (X, Y, Z) of feature point Q are calculated by substituting the rotation matrix R and translation vector T derived from the fundamental matrix E into equation (13).

[0186] As explained above, according to the second variation, the camera three-dimensional coordinates of feature points Q of the subject 58 are calculated based on feature points q1 in the first image and q2 in the second image, which are contained as images between multiple frame images PD. The multiple frame images PD are obtained by capturing images of the subject 58 from different positions using camera 32B. Therefore, according to this structure, the camera three-dimensional coordinates of feature points Q of the subject 58 can be calculated without using LiDAR 32A.

[0187] [Other variations]

[0188] In the above embodiment, a triangular image block 54A is exemplified, but the technology of the present invention is not limited thereto. A quadrilateral image block defined by a quadrilateral grid can also be used instead of the triangular image block 54A. Furthermore, a polygonal image block other than a triangle or quadrilateral can also be used instead of the triangular image block 54A. Thus, the technology of the present invention utilizes polygonal image blocks, thereby enabling easier texture mapping compared to the case where polygonal image blocks are not used.

[0189] In addition, polygonal image blocks can be planar image blocks or curved image blocks. Compared with curved image blocks, planar image blocks have a smaller computational load in texture mapping. Therefore, polygonal image blocks are preferably planar image blocks.

[0190] In the above embodiments, examples of the texture mapping processing program 52 being stored in the storage unit 42 have been described, but the technology of the present invention is not limited thereto. For example, the texture mapping processing program 52 may be stored in a portable storage medium 100 such as an SSD or USB memory. The storage medium 100 is a non-transitory computer-readable storage medium. The texture mapping processing program 52 stored in the storage medium 100 is installed in the computer 39 of the information processing device 20. The processor 40 performs texture mapping processing according to the texture mapping processing program 52.

[0191] Furthermore, the texture mapping processing program 52 can be stored in the storage device of other computers or server devices connected to the information processing device 20 via the network, and can be downloaded and installed on the computer 39 upon request from the information processing device 20.

[0192] In addition, it is not necessary to store the entire texture mapping process 52 in the storage device or storage device 42 of other computers or server devices connected to the information processing device 20; a part of the texture mapping process 52 may also be stored.

[0193] and, Figure 3 The information processing device 20 shown has a built-in computer 39, but the technology of the present invention is not limited thereto. For example, the computer 39 may also be located outside the information processing device 20.

[0194] In the above embodiments, a computer 39 is illustrated, but the technology of the present invention is not limited thereto, and devices including ASICs, FPGAs, and / or PLDs can also be used instead of computer 39. Furthermore, a combination of hardware and software structures can also be used instead of computer 39.

[0195] As the hardware resource for performing the texture mapping processing described in the above embodiments, various processors as shown below can be used. For example, a general-purpose processor, i.e., a CPU, can be used as a hardware resource to perform texture mapping processing by executing software, i.e., a program. Furthermore, processors with dedicated circuitry, such as FPGAs, PLDs, or ASICs, that have circuit structures specifically designed for performing particular processing can be used. Memory is built into or connected to any processor, and any processor performs texture mapping processing using memory.

[0196] The hardware resources for performing texture mapping processing can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware resources for performing texture mapping processing can be a single processor.

[0197] As an example of a single processor, firstly, there is a method where a processor is constructed by combining one or more CPUs and software, with the processor functioning as a hardware resource for performing texture mapping processing. Secondly, there is a method, such as SoC (System-on-a-Chip), where a processor, implemented by a single IC chip, performs the functions of an entire system including multiple hardware resources for performing texture mapping processing. In this way, texture mapping processing is implemented using one or more of the aforementioned processors as hardware resources.

[0198] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the texture mapping process described above is merely one example. Therefore, it is certainly possible to remove unnecessary steps, add new steps, or replace the processing order without departing from the intended purpose.

[0199] The descriptions and illustrations shown above are detailed explanations of the parts related to the technology of this invention, and are merely one example of the technology of this invention. For example, the descriptions related to the structure, function, effect, and effect described above are examples of the structure, function, effect, and effect of the parts related to the technology of this invention. Therefore, it is beyond doubt that unnecessary parts can be deleted, new elements can be added, or replacements can be made to the descriptions and illustrations shown above without departing from the technical spirit of this invention. Furthermore, in order to avoid complexity and facilitate understanding of the parts related to the technology of this invention, descriptions related to technical common sense that do not require special explanation in the implementation of the technology of this invention have been omitted from the descriptions and illustrations shown above.

[0200] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be just A, just B, or a combination of A and B. Furthermore, in this specification, when "and / or" is used to describe three or more items in combination, the same concept as "A and / or B" applies.

[0201] With respect to all the documents, patent applications and technical specifications set forth in this specification, the same documents, patent applications and technical specifications which are specifically set forth and incorporated by reference are also incorporated by reference in this specification.

Claims

1. An image processing apparatus comprising: Processor; and The memory is connected to or built into the processor. The positional relationship between the three-dimensional ranging sensor of the image processing device and the camera device with a higher sampling resolution than the three-dimensional ranging sensor is known, wherein, The processor performs the following processing: Based on the distance measurement results of the three-dimensional ranging sensor for multiple measurement points, obtain the three-dimensional coordinates of the three-dimensional ranging sensor system that can determine the position of the multiple measurement points and are defined by the three-dimensional coordinate system applicable to the three-dimensional ranging sensor. Based on the three-dimensional coordinates of the three-dimensional ranging sensor system, obtain the three-dimensional coordinates of the camera device system as defined by the three-dimensional coordinate system applicable to the camera device; The three-dimensional coordinates of the camera system are converted into two-dimensional coordinates of the camera system that can determine the position within the camera image obtained by the camera device; The three-dimensional coordinates of the camera system are converted into two-dimensional coordinates of the display system, which can determine the position within the image; and The pixels constituting the captured image are assigned to the interpolation positions within the frame based on an interpolation method determined using the two-dimensional coordinates of the camera device system and the two-dimensional coordinates of the display system. The processor performs the following processing: The three-dimensional coordinates of the three-dimensional ranging sensor system are converted into the three-dimensional coordinates of the camera device system, thereby obtaining the three-dimensional coordinates of the camera device system. The processor generates polygonal image blocks based on the three-dimensional coordinates of the camera device system. The three-dimensional coordinates of the camera system define the positions of the intersection points of the polygonal image blocks. The interpolation position is the position within the image that corresponds to a location other than the intersection point of the polygonal image block. The processor performs the following processing: it further allocates the pixels in the multiple pixels contained in the camera image that correspond to the positions of the intersection points of the polygonal image blocks to the positions of the intersection points of the polygonal image blocks within the frame, thereby creating a three-dimensional image.

2. The image processing apparatus according to claim 1, wherein, The polygonal image blocks are defined by a triangular grid or a quadrilateral grid.

3. The image processing apparatus according to claim 1 or 2, wherein, The processor performs the following processing: calculating the three-dimensional coordinates of the camera system based on feature points of the subject contained as images between multiple frames, the multiple frames being obtained by the camera device capturing the subject from different positions.

4. The image processing apparatus according to claim 1 or 2, wherein, The position within the screen refers to the position within the screen displayed on the monitor.

5. The image processing apparatus according to claim 1 or 2, wherein, The memory stores the corresponding association information obtained by establishing a correspondence between the three-dimensional coordinates of the three-dimensional ranging sensor system and the two-dimensional coordinates of the camera device system.

6. The image processing apparatus according to claim 5, wherein, The processor performs the following processing: referring to the corresponding association information, it assigns the pixels constituting the camera image to the interpolation positions.

7. An image processing method, comprising the following steps: Given that the positional relationship between the three-dimensional ranging sensor and the camera device with a higher sampling resolution than the three-dimensional ranging sensor is known, based on the ranging results of the three-dimensional ranging sensor for multiple measuring points, the three-dimensional coordinates of the three-dimensional ranging sensor system that can determine the position of the multiple measuring points are obtained and are defined by the three-dimensional coordinate system applicable to the three-dimensional ranging sensor. Based on the three-dimensional coordinates of the three-dimensional ranging sensor system, obtain the three-dimensional coordinates of the camera device system as defined by the three-dimensional coordinate system applicable to the camera device; The three-dimensional coordinates of the camera system are converted into two-dimensional coordinates of the camera system that can determine the position within the camera image obtained by the camera device; The three-dimensional coordinates of the camera device system are converted into two-dimensional coordinates of the display system, which can determine the position within the image. The pixels constituting the captured image are assigned to the interpolation positions within the frame based on an interpolation method determined by the interpolation method using the two-dimensional coordinates of the camera device system and the two-dimensional coordinates of the display system. The three-dimensional coordinates of the three-dimensional ranging sensor system are converted into the three-dimensional coordinates of the camera device system, thereby obtaining the three-dimensional coordinates of the camera device system; and Polygonal image blocks are generated based on the three-dimensional coordinates of the camera device system. The three-dimensional coordinates of the camera system define the positions of the intersection points of the polygonal image blocks. The interpolation position is the position within the image that corresponds to a location other than the intersection point of the polygonal image block. The pixels in the multiple pixels contained in the camera image that correspond to the positions of the intersection points of the polygonal image blocks are further distributed to the positions of the intersection points of the polygonal image blocks within the frame, thereby creating a three-dimensional image.

8. A storage medium storing a program for causing a computer to perform a process comprising the following steps: Given that the positional relationship between the three-dimensional ranging sensor and the camera device with a higher sampling resolution than the three-dimensional ranging sensor is known, based on the ranging results of the three-dimensional ranging sensor for multiple measuring points, the three-dimensional coordinates of the three-dimensional ranging sensor system that can determine the position of the multiple measuring points are obtained and are defined by the three-dimensional coordinate system applicable to the three-dimensional ranging sensor. Based on the three-dimensional coordinates of the three-dimensional ranging sensor system, obtain the three-dimensional coordinates of the camera device system as defined by the three-dimensional coordinate system applicable to the camera device; The three-dimensional coordinates of the camera system are converted into two-dimensional coordinates of the camera system that can determine the position within the camera image obtained by the camera device; The three-dimensional coordinates of the camera device system are converted into two-dimensional coordinates of the display system, which can determine the position within the image. The pixels constituting the image are assigned to the interpolation positions within the frame based on an interpolation method using the two-dimensional coordinates of the camera device system and the two-dimensional coordinates of the display system. The three-dimensional coordinates of the three-dimensional ranging sensor system are converted into the three-dimensional coordinates of the camera device system, thereby obtaining the three-dimensional coordinates of the camera device system; and Polygonal image blocks are generated based on the three-dimensional coordinates of the camera device system. The three-dimensional coordinates of the camera system define the positions of the intersection points of the polygonal image blocks. The interpolation position is the position within the image that corresponds to a location other than the intersection point of the polygonal image block. The pixels in the multiple pixels contained in the camera image that correspond to the positions of the intersection points of the polygonal image blocks are further distributed to the positions of the intersection points of the polygonal image blocks within the frame, thereby creating a three-dimensional image.

Images