Information processing device, information processing method, and program

Abstract

The information processing apparatus includes a control unit. The control unit acquires data of a first image and data of a second image generated by the imaging device in different postures. The control unit acquires, on the basis of the amount of change in the posture of the imaging device, the amount of change in the posture of the distance measuring device in conjunction with the posture of the imaging device, said amount of change in the posture of the imaging device being calculated on the basis of the position of the same subject in the first image and the position of the same subject in the second image.

Description

[0001] Cross-reference to related applications

[0002] This application claims the priority of Japanese Patent Application No. 2022-129066 filed in Japan on August 12, 2022, and the entire disclosure of that application is incorporated herein by reference. Technical Field

[0003] The present invention relates to an information processing device, an information processing method and a program. Background Art

[0004] Conventionally, there is known a technique for generating an environment map using data acquired by a distance measuring device such as LiDAR (Light Detection And Ranging) (see Patent Document 1).

[0005] Prior art literature

[0006] Patent document 1: Japanese Patent Application Publication No. 2020-86678. Summary of the invention

[0007] An information processing device according to one embodiment of the present invention includes:

[0008] The control unit acquires data of a first image and a second image generated by the camera in different postures,

[0009] The control unit obtains the change in posture of the ranging device linked to the posture of the shooting device based on the change in posture of the shooting device, and the change in posture of the shooting device is calculated based on the position of the same subject in the first image and the position in the second image.

[0010] An information processing method according to an embodiment of the present invention includes:

[0011] Acquire data of a first image and data of a second image generated by a camera in different postures,

[0012] According to the change in the posture of the shooting device, the change in the posture of the ranging device linked to the posture of the shooting device is obtained, and the change in the posture of the shooting device is calculated based on the position of the same subject in the first image and the position in the second image.

[0013] A program according to an embodiment of the present invention,

[0014] Causes a computer to perform the following steps:

[0015] Acquire data of a first image and data of a second image generated by a camera in different postures,

[0016] According to the change in the posture of the shooting device, the change in the posture of the ranging device linked to the posture of the shooting device is obtained, and the change in the posture of the shooting device is calculated based on the position of the same subject in the first image and the position in the second image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a block diagram of a generation system according to one embodiment of the present invention.

[0018] Figure 2 It is a diagram for explaining the template matching process.

[0019] Figure 3 This is a diagram for explaining the setting of a template image.

[0020] Figure 4 This is a flowchart showing an example of the sequence of the information processing method. DETAILED DESCRIPTION

[0021] A technology for estimating the posture information of a distance measuring device with high accuracy is expected. For example, the posture information of the distance measuring device is used to generate an environment map. If the posture information of the distance measuring device can be estimated with high accuracy, the environment map can be generated with high accuracy. According to one embodiment of the present invention, a technology for estimating the posture information of a distance measuring device with high accuracy can be provided.

[0022] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0023] (Generation system configuration)

[0024] Figure 1 The generation system 1 shown can generate an environment map. The generation system 1 is mounted on a mobile object, for example. The mobile object is, for example, a car, a robot, or a person. However, the mobile object is not limited to these.

[0025] The generation system 1 includes a distance measuring device 10, a photographing device 20, and an information processing device 30. The distance measuring device 10 and the photographing device 20 can communicate with the information processing device 30 via a communication line. The information processing device 30 may not be mounted on a mobile body. In this case, the information processing device 30 is configured to be able to communicate with the distance measuring device 10 and the photographing device 20 mounted on the mobile body.

[0026] The posture of the distance measuring device 10 is linked to the posture of the camera 20. As an example, the distance measuring device 10 and the camera 20 may be fixed to each other. By fixing the distance measuring device 10 and the camera 20 to each other, the posture of the distance measuring device 10 and the posture of the camera 20 can be linked. In addition, by fixing the distance measuring device 10 and the camera 20 to each other, the relative position and posture between the distance measuring device 10 and the camera 20 can be determined. The distance measuring device 10 and the camera 20 may be fixed to the moving object in such a manner that the relative position and posture are fixed to each other.

[0027] The distance measuring device 10 is composed of, for example, LiDAR. The distance measuring device 10 transmits electromagnetic waves to the actual space around the distance measuring device 10. The electromagnetic waves emitted by the distance measuring device 10 are reflected by the object and return to the distance measuring device 10 as reflected waves. The distance measuring device 10 detects the electromagnetic waves reflected by the object among the emitted electromagnetic waves, i.e., the reflected waves. The distance measuring device 10 measures the distance to the object by directly measuring the time from the emission of the electromagnetic wave to the detection of the reflected wave of the emitted electromagnetic wave.

[0028] The distance measuring device 10 uses electromagnetic waves to scan the actual space around the distance measuring device 10 while detecting the reflected waves of the electromagnetic waves, thereby generating point cloud data (position data). The point cloud data represents the position of the object around the distance measuring device 10. The point cloud data is, for example, a collection of points representing the position of the object in the three-dimensional real space. The point cloud data may include the position information of the object in the three-dimensional space and the distance information from the distance measuring device 10 to the object. The position information of the object in the three-dimensional space in the point cloud data is obtained by a local coordinate system based on the position of the distance measuring device 10. In this embodiment, the local coordinate system is an XYZ coordinate system composed of mutually orthogonal X-axis directions, Y-axis directions, and Z-axis directions. It is assumed that the position information of the local coordinate system is represented by (X, Y, Z). In this embodiment, the origin of the local coordinate system is the position of the distance measuring device 10. In this embodiment, the point cloud data includes the position information (X, Y, Z) of the local coordinate system of the object and the distance information from the position of the distance measuring device 10, i.e., the origin of the local coordinate system, to the object. However, the point cloud data may include only one of the position information of the local coordinate system of the object and the distance information from the ranging device 10 to the object. The point cloud data may include, for example, one frame of distance information. One frame of distance information can be obtained by scanning the measurement range described later. The ranging device 10 may generate point cloud data at any frame rate. If the ranging device 10 generates point cloud data, the generated point cloud data is sent to the information processing device 30.

[0029] Hereinafter, the range in actual space where the distance measuring device 10 transmits electromagnetic waves to generate point cloud data is also referred to as a “measurement range”.

[0030] The photographing device 20 has a photographing optical system and a photographing element. The photographing device 20 can generate an image by photographing the actual space around the photographing device 20. In the image, a two-dimensional coordinate system is determined. In the present embodiment, the two-dimensional coordinate system set for the image is an xy coordinate system consisting of mutually orthogonal x-axis directions and y-axis directions. The x-axis direction corresponds to the horizontal direction. The horizontal direction is a direction perpendicular to the vertical direction. The y-axis direction corresponds to a direction perpendicular to the horizontal direction.

[0031] The camera 20 may generate images at any frame rate. The frame rate of the camera 20 may be the same as or different from the frame rate of the distance measuring device 10. When the camera 20 generates an image, the generated image is sent to the information processing device 30.

[0032] Hereinafter, the range in the actual space that the camera 20 takes pictures to generate an image is also referred to as a "shooting range". This shooting range partially overlaps with the ranging range of the ranging device 10. However, the shooting range of the camera 20 may not overlap with the measuring range of the ranging device 10. Alternatively, the entire shooting range of the camera 20 may overlap with the ranging range of the ranging device 10, or the entire ranging range of the ranging device 10 may overlap with the shooting range of the camera 20.

[0033] The information processing device 30 is, for example, a computer. The information processing device 30 may be any computer such as a general-purpose computer, a workstation, or a dedicated computer.

[0034] The information processing device 30 includes a communication unit 31 , an output unit 32 , a storage unit 33 , and a control unit 34 .

[0035] The communication unit 31 is configured to include at least one communication module capable of communicating with the distance measuring device 10 and the imaging device 20 via a communication line. The communication module is a communication module that complies with the standard of the communication line. The communication line includes at least one of wired and wireless.

[0036] The communication unit 31 may be configured to include at least one communication module capable of communicating with other systems of the mobile body equipped with the generation system 1. The communication module is a communication module that complies with a communication standard between the information processing device 30 and other systems.

[0037] The output unit 32 can output data. The output unit 32 is configured to include at least one output interface capable of outputting data. The output interface is, for example, a display. The output unit 32 may also be configured to include a device capable of outputting information to an external storage medium.

[0038] The storage unit 33 is configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of them. The semiconductor memory is, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). The RAM is, for example, an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory). The ROM is, for example, an EEPROM (Electrically Erasable Programmable Read Only Memory). The storage unit 33 can function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 33 stores data used for the operation of the information processing device 30 and data obtained by the operation of the information processing device 30.

[0039] The controller 34 is configured to include at least one processor, at least one dedicated circuit, or a combination thereof. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) or a dedicated processor dedicated to a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 34 controls each unit of the information processing device 30 and performs processing related to the operation of the information processing device 30.

[0040] The control unit 34 generates an environment map using the point cloud data generated by the distance measuring device 10. The control unit 34 acquires the posture of the distance measuring device 10 to generate the environment map. The control unit 34 calculates the movement amount of the subject from the point cloud data generated by the distance measuring device 10, thereby being able to acquire the posture of the distance measuring device 10. However, for example, when the moving object is moving on a snowy field, there may be a case where there is no concave or convex object within the measurement range of the distance measuring device 10. In this case, it is highly likely that the point cloud data generated by the distance measuring device 10 does not include a characteristic portion. The characteristic portion of the point cloud data generated by the distance measuring device 10 refers to a point cloud having a shape that allows point clouds to match each other. If the point cloud data does not include a characteristic portion, the control unit 34 cannot estimate the posture of the distance measuring device 10 with high accuracy. Here, the image generated by the imaging device 20 includes information such as brightness, and can include more characteristic portions than the point cloud data generated by the distance measuring device 10. The characteristic portion of the image generated by the imaging device 20 refers to a portion of the subject portion in the image that is clearly distinguished from other adjacent portions and can be determined by image processing. Therefore, the control unit 34 performs image processing on the image generated by the camera 20, thereby calculating the amount of change in the posture of the camera 20. As described above, the posture of the distance measuring device 10 is linked to the posture of the camera 20. Therefore, by calculating the amount of change in the posture of the camera 20, the control unit 34 can obtain the amount of change in the posture of the distance measuring device 10 linked to the posture of the camera 20. Next, the generation process of the environment map of this embodiment is described.

[0041] The control unit 34 obtains the data of the first image and the data of the second image generated by the camera 20 in different postures. In the present embodiment, the control unit 34 receives the data of the first image and the data of the second image generated at different times from the camera 20 using the communication unit 31. The data of the first image and the data of the second image generated at different times become the data of the first image and the data of the second image generated by the camera 20 in different postures when the mobile body equipped with the camera 20 moves. That is, the control unit 24 obtains the data of the first image and the data of the second image generated by the camera 20 in different postures by receiving the data of the first image and the data of the second image generated at different times. In the present embodiment, the control unit 34 continuously receives the data of the image from the camera 20 at a preset time interval using the communication unit 31, thereby receiving the data of the first image and the data of the second image. The time interval can be set according to the frame rate of the camera 20. For example, the time interval can be the inverse of the frame rate of the camera 20. Alternatively, the time interval can be set according to the average speed of the mobile body equipped with the generation system 1. For the processing described later, the control unit 34 receives the point cloud data generated at the same time as the first image and the second image using the communication unit 31. In addition, in the present embodiment, the point cloud data generated within the specified time before and after the time when the first image and the second image are generated can be regarded as the point cloud data generated at the same time. In the present embodiment, the control unit 34 receives the image data from the camera 20 at the above time intervals and receives the point cloud data from the distance measuring device 10, thereby receiving the point cloud data generated at the same time as the first image and the second image. Here, since the camera 20 sends the generated image to the information processing device 30 as soon as it generates the image, the time when the image is received from the camera 20 can also be regarded as the time when the camera 20 generates the image. Since the distance measuring device 10 sends the point cloud data to the information processing device 30 as soon as it generates the point cloud data, the time when the point cloud data is received from the distance measuring device 10 can also be regarded as the time when the distance measuring device 10 generates the point cloud data.

[0042] In the following, it is assumed that the first image is taken at time T N-1 (N is an integer greater than 2) to generate an image. The first image is also referred to as the first image (T N-1 ). Assume that the second image is taken at time T N (N is an integer greater than or equal to 2). The second image is also referred to as the second image (T N ). Time T N and time T N-1 The time interval between them is the above-mentioned pre-set time interval.

[0043] The control unit 34 calculates the amount of change in the posture of the imaging device 20 based on the position of the same subject in the first image and the position in the second image, thereby acquiring the amount of change in the posture of the distance measuring device 10 .

[0044] In this embodiment, the control unit 34 generates a first image (T N-1 ) in the second image (T N ) and calculates the change amount of the imaging device 20. The control unit 34 obtains the offset dx N and the offset dy N As the same subject in the first image (T N-1 ) in the second image (T N ) The offset dx N is the x-axis direction of the subject in the first image (T N-1 ) in the second image (T N ). For example, the offset dx N is given as a number of pixels. The offset dy N is the position of the subject in the first image (T N-1 ) in the second image (T N ). For example, the offset dy N is given as a number of pixels.

[0045] The control unit 34 processes the first image (T N-1 ) and the second image (T N ) Perform template matching processing to obtain the offset dx N and the offset dy N However, the control unit 34 may also obtain the offset dx by any processing. N and the offset dy N As another example, the control unit 34 may N-1 ) and the second image (T N ) Perform feature point matching to obtain the offset dx N and the offset dy N .

[0046] In the template matching process, the control unit 34 converts the first image (T N-1 ) as a part of the template image. The control unit 34 calculates the template image and the second image (T N) in the image portion that overlaps with the template image. The control unit 34 may calculate the correlation value by any algorithm such as ZNCC (Zero means Normalized Cross Correlation). In this embodiment, the higher the correlation value, the closer the template image is to the second image (T N ) is higher. When the correlation value exceeds the correlation threshold, the control unit 34 detects the partial image as the second image (T N ) is similar to the template image. The relevant threshold can be pre-set based on the data of past images. The control unit 34 N-1 ) and the xy coordinates of the template image in the second image (T N ) is the difference in xy coordinates of the part of the image similar to the template image, and the offset dx is obtained. N and the offset dy N For example, the control unit 34 calculates the first image (T N-1 ) and the xy coordinates of the center of the template image in the second image (T N ) is the difference in the xy coordinates of the center of the partial image similar to the template image, and the calculated difference is obtained as the offset dx N and the offset dy N However, when there are multiple correlation values ​​exceeding the correlation threshold, the control unit 34 may select the largest correlation value among the multiple correlation values ​​exceeding the correlation threshold, and use the offset dx corresponding to the selected correlation value. N and the offset dy N In addition, the control unit 34 may select the largest correlation value from the calculated multiple correlation values, and when the selected correlation value exceeds the correlation threshold, use the offset dx corresponding to the selected correlation value. N and the offset dy N .

[0047] For example, the control unit 34 receives information such as Figure 2 The data of the first image 40 and the data of the second image 41 are shown. The first image 40 and the second image 41 respectively capture the scenery of the mountain and the snowfield near the moving object. The first image 40 includes a template image 42. In the second image 41, for convenience, the position of the template image 42 is indicated by a dotted line. The control unit 34 detects a partial image 43 similar to the template image 42 in the second image 41. The control unit 34 obtains the offset dx according to the difference between the xy coordinates of the template image 42 and the xy coordinates of the partial image 43. N and the offset dy N .

[0048] The template image may be an image of a pre-set area in the area of ​​the first image. As an example, the area corresponding to the far side of the camera 20 in the area of ​​the first image may be pre-set as the area of ​​the template image. The area corresponding to the far side of the camera 20 in the first image is an area where the possibility of capturing scenery such as mountains becomes higher than that of the area corresponding to the near side of the camera 20 in the first image. Based on the data of the image generated in the past, the area where the possibility of capturing scenery such as mountains in the first image is high can be determined, and the determined area can be pre-set as the area of ​​the template image. For example, in the case where the moving body is moving on the snowfield, it is highly likely that only snow is captured in the area corresponding to the near side of the camera 20 in the first image. In this case, it is highly likely that the feature part is not included in the area corresponding to the near side of the camera 20 in the first image. In contrast, even in the case where the moving body is moving on the snowfield, it is highly likely that scenery such as mountains is captured in the area corresponding to the far side of the camera 20 in the first image. It is highly likely that the part of the first image where scenery such as mountains is captured is the feature part. Therefore, by presetting the area of ​​the first image corresponding to the far side of the camera 20 as the area of ​​the template image, the possibility of including the feature part in the template image becomes higher. With such a structure, the time for the control unit 34 to detect the image part similar to the template image in the second image can be reduced through the template matching process. In addition, with such a structure, the control unit 34 can detect the image part similar to the template image in the second image with high accuracy through the template processing.

[0049] The template image may also be a partial image selected by the control unit 34 in the first image. In this case, the control unit 34 may determine the partial image corresponding to the far side of the camera 20 from the first image based on the point cloud data generated at the same time as the first image. The control unit 34 may select the determined partial image as the template image. That is, the control unit 34 may set the determined partial image as the template image. The partial image corresponding to the far side of the camera 20 in the first image is a partial image that is more likely to capture scenery such as mountains than the partial image corresponding to the near side of the camera 20 in the first image. As an example, when at least a part of the shooting range of the camera 20 overlaps with the measurement range of the distance measuring device 10, the control unit 34 may determine the area in the area of ​​the first image where the distance to the object cannot be measured by the distance measuring device 10 (the range beyond the predetermined distance from the camera 20) using the point cloud data generated at the same time as the first image. If the distance from the distance measuring device 10 to the object exceeds the range where the distance can be measured by the distance measuring device 10, the distance to the object cannot be measured by the distance measuring device 10. That is, the area where the distance to the object cannot be measured by the distance measuring device 10 is likely to be on the far side of the imaging device 20. The control unit 34 may set a partial image of the area corresponding to the far side of the imaging device 20 in the first image as a template image, compared to the area where the distance to the object cannot be measured by the distance measuring device 10.

[0050] For example, in Figure 3 In the first image 40, a portion of the shooting range of the shooting device 20 overlaps with the measuring range of the distance measuring device 10. The range 44 included in the first image 40 is a range corresponding to the measuring range of the distance measuring device 10. In the range 44, the shaded area 45 is an area where the distance to the object can be measured using the distance measuring device 10. In the range 44, the unshaded area 46 is an area where the distance to the object cannot be measured using the distance measuring device 10. Compared with the area 45, the control unit 34 sets the partial image in the area 47 corresponding to the far side of the shooting device 20 in the first image as a template image. Compared with the area 45, the template image 48 is an example of a partial image of the area 47 corresponding to the far side of the shooting device 20.

[0051] The size of the template image can be pre-set according to the size of the subject in the first image, etc. Alternatively, the size of the template image can be set by the control unit 34. In this case, the control unit 34 can obtain the speed information of the distance measuring device 10, and set the size of the template image according to the obtained speed information. In this embodiment, the speed information of the distance measuring device 10 is the speed information of the moving body equipped with the distance measuring device 10. The control unit 34 can be set so that the faster the speed of the distance measuring device 10, that is, the speed of the moving body in this embodiment, the smaller the size of the template image. The control unit 34 can obtain the speed information of the moving body according to the speed sensor of the moving body. The control unit 34 can also use the LiDAR odometry technology to estimate the movement amount of the moving body according to the measurement data received from the distance measuring device 10, and obtain the speed information of the moving body according to the estimated movement amount. The control unit 34 can also use the visual odometry technology to estimate the movement amount of the moving body according to the change of the image received from the shooting device 20 at a preset time interval, and obtain the speed information of the moving body according to the estimated movement amount. The control unit 34 may also communicate with other systems of the moving body via the communication unit 31 to receive the speed information of the moving body. By setting the size of the template image according to the speed information of the distance measuring device 10, the control unit 34 can detect an image portion similar to the template image in the template matching process. For example, since the size of the template image is smaller as the speed of the moving body is faster, the control unit 34 can detect an image portion similar to the template image in the template matching process even if the speed of the moving body is faster.

[0052] If you get the offset dx N and the offset dy N , the control unit 34 calculates the rotation angle θ of the camera 20 YN and the rotation angle θ PN As the amount of change in the posture of the imaging device 20 .

[0053] Rotation angle θ YN is the camera 20 at time T N The rotation angle θ Y , which is based on the rotation angle θ of the camera 20 at the first moment Y The rotation angle θ Y The rotation angle θ is the rotation angle around the vertical axis. Y The first moment corresponds to the roll angle, pitch angle, and yaw angle of the object. As described later, the first moment is used to set the world coordinate system. In this embodiment, the first moment is set to time T1. However, the first moment can be set arbitrarily.

[0054] Rotation angle θ PN is the camera 20 at time TN The rotation angle θ P , which is based on the rotation angle θ of the camera 20 at the first moment, that is, the moment T1 P The rotation angle θ P The rotation angle θ is the rotation angle around the horizontal axis. P Corresponds to the pitch angle among the roll angle, pitch angle and yaw angle that represent the attitude of the object.

[0055] The control unit 34 calculates the rotation angle θ of the imaging device 20 using the following equations (1) and (2): YN and the rotation angle θ PN .

[0056] θ YN =dx N ×(Hdeg / Hpixel)+θ Y(N-1) Formula (1)

[0057] θ PN =dy N ×(Vdeg / Vpixel)+θ P(N-1) Formula (2)

[0058] In equation (1), the horizontal viewing angle Hdeg is the horizontal viewing angle of the camera 20. The number of pixels Hpixel is the number of pixels arranged along the x-axis direction in the first image. When N=2, the right side θ Y(N-1) That is, θ Y1 is zero.

[0059] In equation (2), the vertical viewing angle Vdeg is the vertical viewing angle of the camera 20. The number of pixels Vpixel is the number of pixels arranged along the y-axis direction in the first image. When N=2, the right θ P(N-1) That is, θ P2 is zero.

[0060] If the rotation angle θ of the camera 20 is calculated YN and the rotation angle θ PN , the control unit 34 will be at time T N The position information of the local coordinate system of the object included in the generated point cloud data is converted into the position information of the world coordinate system. The world coordinate system is a coordinate system set according to the actual space outside the distance measuring device 10. In this embodiment, the local coordinate system of the distance measuring device 10 at the first moment, that is, the moment T1, is used as the world coordinate system. N (N is an integer greater than 2) is also recorded as the second time. N The position information (X, Y, Z) of the local coordinate system of the object included in the generated point cloud data is also recorded as position information (X N , Y N , ZN ).

[0061] The control unit 34 uses the following formula (3) to set the N The generated point cloud data includes the position information of the local coordinate system of the object (X N , Y N , Z N ) is converted into the world coordinate system, that is, the position information (X1, Y1, Z1) of the local coordinate system of the ranging device 10 at the first moment.

[0062] [Mathematical formula 1]

[0063]

[0064] The control unit 34 can continuously receive the image and point cloud data from the camera 20 and the distance measuring device 10 at the above time intervals, thereby repeatedly performing the above processing. N The position information of the local coordinate system of the object in the generated point cloud data (X N , Y N , Z N ) is further converted into the position information (X1, Y1, Z1) of the local coordinate system of the distance measuring device 10 at the first moment, so that the environment map can be continuously generated. The control unit 34 can also display the generated environment map on the display of the output unit 32. The control unit 34 can also output the generated environment map to an external storage medium using the output unit 32.

[0065] (Generate system actions)

[0066] Figure 4 3 is a flowchart showing an example of the sequence of the generation process of the environment map. The generation process of the environment map is equivalent to an example of the information processing method of this embodiment. For example, when the mobile body starts to move, the control unit 34 starts the generation process of the environment map. The control unit 34 can communicate with other systems of the mobile body via the communication unit 31 to detect that the mobile body starts to move.

[0067] The control unit 34 receives the first image (T N-1 ) data, and receives the point cloud data from the distance measuring device 10 using the communication unit 31 (step S1). N-1 ) and point cloud data at the same time T N-1 generate.

[0068] The control unit 34 receives the second image (T N ) data, and receives the point cloud data from the distance measuring device 10 using the communication unit 31 (step S2). The second image (TN ) and point cloud data at the same time T N generate.

[0069] The control unit 34 processes the first image (T N-1 ) and the second image (T N ) performs template matching processing (step S3). The control unit 34 obtains the offset dx by performing template matching processing. N and the offset dy N (Step S4).

[0070] The control unit 34 uses the offset dx obtained in the process of step S4 N and offset dy N The rotation angle θ of the camera 20 is calculated by using equations (1) and (2): YN and the rotation angle θ PN (Step S5).

[0071] The control unit 34 uses the rotation angle θ of the imaging device 20 calculated in the process of step S5. YN and the rotation angle θ PN The control unit 34 generates an environment map by converting the position information of the local coordinate system of the object included in the point cloud data received in the process of step S2 into the position information of the world coordinate system.

[0072] The control unit 34 receives the second image (T N ) is set as the new first image (T N-1 ) (Step S7). After the process of step S7, the control unit 34 executes the process of step S2 again. In the process of step S2 executed again, the control unit 34 receives a new second image (T N ).

[0073] In the process of the above step S3, each time the template matching process is performed, the control unit 34 may select a template image from the first image to set it. Alternatively, the control unit 34 may select and set a template image from the first image when the set condition is satisfied. The set condition may be a condition that the above correlation value is equal to or less than the correlation threshold. In this case, if the correlation value is equal to or less than the correlation threshold, the control unit 34 may determine the partial image corresponding to the far side of the camera 20 from the first image based on the point cloud data generated at the same time as the first image. If the correlation values ​​in the entire second image are equal to or less than the correlation threshold, the control unit 34 may determine the partial image corresponding to the far side of the camera 20 from the first image based on the point cloud data generated at the same time as the first image. The control unit 34 may set the determined partial image as the template image. If a new template image has been set, the control unit 34 may also perform the template matching process again. With such a structure, in the template process, the control unit 34 can determine the image portion in the second image that is similar to the template image with high accuracy.

[0074] For example, when the moving body stops, the control unit 34 ends Figure 4 The control unit 34 can detect that the moving object has stopped moving by communicating with other systems of the moving object via the communication unit 31 .

[0075] Thus, in the information processing device 30 of the present embodiment, the control unit 34 calculates the amount of change in the posture of the camera 20 based on the position of the same subject in the first image and the position in the second image. The control unit 34 obtains the amount of change in the posture of the distance measuring device 10 linked to the posture of the camera 20 by calculating the amount of change in the posture of the camera 20. As described above, for example, when the moving object is moving on a snowy field, there may be a case where there is no concave or convex object within the measurement range of the distance measuring device 10. In this case, since it is highly likely that the two-dimensional image extracted from the point cloud data does not include a feature portion, the control unit 34 cannot accurately estimate the posture of the distance measuring device 10 by performing image processing on the extracted two-dimensional image. In the present embodiment, the control unit 34 obtains the amount of change in the posture of the distance measuring device 10 linked to the posture of the camera 20 by calculating the amount of change in the posture of the camera 20 using the first image and the second image generated by the camera 20. With such a configuration, even if there is no uneven object within the measurement range of the distance measuring device 10, the amount of change in the posture of the distance measuring device 10 can be estimated with high accuracy. Therefore, according to this embodiment, a technique for estimating the posture information of the distance measuring device 10 with high accuracy can be provided.

[0076] Furthermore, in the present embodiment, the control unit 34 may receive the point cloud data corresponding to the first image from the distance measuring device 10 using the communication unit 31, thereby acquiring the point cloud data. The point cloud data corresponding to the first image may be point cloud data generated at the same time as the first image. The point cloud data corresponding to the first image may also be point cloud data generated by the distance measuring device 10 when the camera 20 is in the posture for generating the first image. The control unit 34 may set a predetermined distance based on the point cloud data. The predetermined distance may be a distance of the point cloud data that the distance measuring device 10 cannot acquire. The control unit 34 may calculate the amount of change in the posture of the camera 20 based on the position of the subject in the range exceeding the predetermined distance from the camera 20 in the first image and the position in the second image. When the control unit 34 performs the template matching process, as described above, the partial image corresponding to the far side may be set as the template image, thereby detecting the subject in the range exceeding the predetermined distance from the camera 20. Here, it is highly likely that the partial image corresponding to the near side of the camera 20 in the first image does not include a characteristic portion. For example, when a moving body is moving on a snowy field, there is a high possibility that only snow is captured in the partial image corresponding to the near side of the camera 20 in the first image. In contrast, there is a high possibility that the partial image corresponding to the far side of the camera 20 in the first image includes a characteristic portion. For example, even when a moving body is moving on a snowy field, there is a high possibility that scenery such as mountains is captured in the partial image corresponding to the far side of the camera 20 in the first image. There is a high possibility that the portion of the first image where scenery such as mountains is captured becomes a characteristic portion. Therefore, the partial image corresponding to the far side of the camera 20 in the area of ​​the first image is set as a template image, thereby increasing the possibility that the template image includes a characteristic portion. With such a structure, the time for the control unit 34 to detect an image portion similar to the template image in the second image can be reduced through template matching processing. In addition, with such a structure, the control unit 34 can detect an image portion similar to the template image in the second image with high accuracy through template processing.

[0077] In addition, in the present embodiment, if the correlation between the image of the subject in the range exceeding the predetermined distance and the first image and the second image exceeds the predetermined value, the control unit 34 can calculate the amount of change in the posture of the camera 20 based on the position of the subject in the range exceeding the predetermined distance in the first image and the position in the second image. When the template matching process is executed, the control unit 34 can calculate the above correlation value as a value indicating the correlation between the image of the subject in the range exceeding the predetermined distance and the second image. If the calculated correlation value is equal to or less than the correlation threshold value as a predetermined value, the control unit 34 can specify the partial image corresponding to the far side of the camera 20 from the first image based on the point cloud data. The control unit 34 can also set the determined partial image as the template image. That is, the control unit 34 can calculate the amount of change in the posture of the camera 20 based on the position of the subject in the range exceeding the predetermined distance in the first image and the position in the second image. With such a configuration, in the template process, the control unit 34 can detect the image portion similar to the template image in the second image with high accuracy.

[0078] In addition, in the present embodiment, the control unit 34 can obtain the first data and the second data by receiving the first data and the second data from the ranging device 10. The first data is the point cloud data generated by the ranging device 10 when the shooting device 20 is in a posture that has generated the first image. The second data is the point cloud data generated by the ranging device 10 when the shooting device 20 is in a posture that has generated the second image. The first data may be point cloud data generated at the same time as the first image, i.e., the first moment. The second data may be point cloud data generated at the same time as the second image, i.e., the second moment. The control unit 34 may also adjust the positional relationship between the position of the object shown in the first data and the position of the object shown in the second data by the acquired change in the posture of the ranging device 10. Furthermore, the control unit 34 may also generate an environmental map based on the position of the object whose positional relationship has been adjusted. As described above, the first moment is T1 and the second moment is T N (N is an integer greater than or equal to 2), the change in the posture of the distance measuring device 10 from the first moment to the second moment may be the rotation angle θ of the imaging device 20. YN and the rotation angle θ PN In this case, the control unit 34 may also control the image pickup device 20 by adjusting the rotation angle θ YN and the rotation angle θ PN The positional relationship between the position of the object shown in the first data and the position of the object shown in the second data can be adjusted by using the equation (3). YN and the rotation angle θPN The position information of the local coordinate system of the object in the point cloud data is converted into the position information of the local coordinate system of the ranging device 10 at the first moment, that is, the position information of the world coordinate system, by using equation (3). The world coordinate system may also be the local coordinate system of the ranging device 10 at the first moment. The control unit 34 may convert the position information of the local coordinate system of the object in the point cloud data into the position information of the world coordinate system by using the change in the posture of the ranging device 10 from the first moment to the second moment, thereby generating an environment map. By setting the world coordinate system to the local coordinate system of the ranging device 10 at the first moment, it is possible to generate an environment map by using a simple coordinate conversion such as equation (3).

[0079] In addition, in the present embodiment, the control unit 34 may acquire speed information of the change in posture of the ranging device 10, and set the size of the image of the subject according to the acquired speed information. The control unit 34 may acquire the speed information of the ranging device 10 as the speed information of the change in posture of the ranging device 10 as described above. The control unit 34 may also set the size of the template image as the size of the image of the subject. The control unit 34 may set the image of the subject to be smaller as the speed of the change in posture of the ranging device 10 is faster. As described above, the control unit 34 may also set the size of the template image to be smaller as the speed of the ranging device 10 is faster. By setting the size of the template image according to the speed information of the ranging device 10, the control unit 34 can detect an image portion similar to the template image in the template matching process. For example, the faster the speed of the ranging device 10 is, the smaller the size of the template image is. Thus, even if the speed of the ranging device 10 is faster, the control unit 34 can detect an image portion similar to the template image in the template matching process.

[0080] In one embodiment, (1) an information processing device includes:

[0081] The control unit acquires data of a first image and a second image generated by the camera in different postures,

[0082] The control unit obtains the change in posture of the ranging device linked to the posture of the shooting device based on the change in posture of the shooting device, and the change in posture of the shooting device is calculated based on the position of the same subject in the first image and the position in the second image.

[0083] (2) In the information processing device of (1) above,

[0084] The control unit acquires position data indicating the position of an object existing around the distance measuring device.

[0085] The control unit calculates a change in the posture of the photographing device based on the position of the subject in the first image and the position of the subject in the second image, which is within a range exceeding a specified distance from the photographing device, and the specified distance is a distance at which the ranging device cannot obtain the position data.

[0086] (3) In the information processing device of (1) or (2) above,

[0087] The control unit acquires the amount of change if the correlation between the first image and the second image of the image of the subject in a range exceeding the predetermined distance exceeds a predetermined value.

[0088] (4) In any one of the information processing devices (1) to (3) above,

[0089] The control unit acquires first data and second data from the distance measuring device, the first data being position data generated by the photographing device in a posture for generating the first image, and the second data being position data generated by the photographing device in a posture for generating the second image,

[0090] The control unit adjusts a positional relationship between a position of the object represented by the first data and a position of the object represented by the second data according to the change amount.

[0091] (5) In any one of the information processing devices (1) to (4) above,

[0092] The control unit generates an environment map based on the positions of the objects after the positional relationship is adjusted.

[0093] (6) In any one of the information processing devices (1) to (5) above,

[0094] The control unit sets the size of the image of the subject according to the speed at which the posture of the distance measuring device changes.

[0095] (7) In any one of the information processing devices (1) to (6) above,

[0096] The control unit sets the image so that the size of the image of the subject becomes smaller as the speed at which the posture of the distance measuring device changes becomes faster.

[0097] (8) In any one of the information processing devices (1) to (7) above,

[0098] The control unit performs template matching processing on the first image and the second image, thereby obtaining a difference between a position of the same subject in the first image and a position of the same subject in the second image.

[0099] In the template matching process, an image portion similar to a portion of the first image set as a template image is detected from the second image.

[0100] In one embodiment, (9) an information processing method,

[0101] include:

[0102] Acquire data of a first image and data of a second image generated by a camera in different postures,

[0103] According to the change in the posture of the shooting device, the change in the posture of the ranging device linked to the posture of the shooting device is obtained, and the change in the posture of the shooting device is calculated based on the position of the same subject in the first image and the position in the second image.

[0104] In one embodiment, (10) a program,

[0105] Make the computer perform the following actions:

[0106] Acquire data of a first image and data of a second image generated by a camera in different postures,

[0107] According to the change in the posture of the shooting device, the change in the posture of the ranging device linked to the posture of the shooting device is obtained, and the change in the posture of the shooting device is calculated based on the position of the same subject in the first image and the position in the second image.

[0108] Although the present invention has been described according to the drawings and embodiments, it should be noted that it is easy for a person skilled in the art to make various deformations or modifications according to the present invention. Therefore, it should be noted that the scope of the present invention includes these deformations or modifications. For example, the functions included in each functional unit can be reconfigured in a logically non-contradictory manner. Multiple functional units can be combined into one or divided. The above-mentioned embodiments of the present invention are not limited to being implemented strictly in accordance with the embodiments described separately, and can be implemented by appropriately combining each feature or omitting a part of the features. That is to say, for a person skilled in the art, the content of the present invention can be variously deformed and modified according to the present invention. Therefore, the scope of the present invention includes these deformations and modifications. For example, in each embodiment, each functional unit, each unit, each step, etc. can be added to other embodiments in a logically non-contradictory manner, or each functional unit, each unit, each step, etc. of other embodiments can be replaced. In addition, in each embodiment, multiple functional units, each unit, each step, etc. can be combined into one or divided. In addition, the above-mentioned embodiments of the present invention are not limited to being implemented strictly in accordance with the embodiments described separately, and can also be implemented by appropriately combining each feature or omitting a part of the features.

[0109] For example, in the above embodiment, the first moment is described as moment T1. However, the first moment is not limited to moment T1. The first moment may also be a moment other than moment T1. As an example, the first moment may also be moment T M+1 (M is an integer greater than or equal to 1). In this case, the second time is time T M+1 Other than the moment.

[0110] For example, in the above-mentioned embodiment, the imaging device 20 is described as having an imaging optical system and an imaging element. However, the structure of the imaging device 20 is not limited thereto. As long as an image with brightness can be generated, that is, as long as an image capable of executing template matching processing can be generated, the imaging device 20 can be of any structure. For example, the imaging device 20 can be an infrared camera.

[0111] For example, the control unit 34 can obtain the time from time T1 to time T of the distance measuring device 10 by the above method. N The control unit 34 can also obtain the movement amount of the distance measuring device 10 from time T1 to time T according to the obtained movement amount. N The control unit 34 may also add the obtained translation vector to the right side of equation (3).

[0112] For example, an embodiment in which a general-purpose computer functions as the above-mentioned information processing device 30 can also be realized. Specifically, a program recording the processing contents of each function of the information processing device 30 that realizes the above-mentioned embodiment is stored in the memory of a general-purpose computer, and the program is read out and executed by a processor. Therefore, the present invention can also be realized as a program executable by a processor or a non-temporary computer-readable medium storing the program.

[0113] In the present invention, the records such as "first" and "second" are identifiers for distinguishing the structure. The structures distinguished by the records such as "first" and "second" in the present invention can exchange the numbers of the structures. For example, the first image can exchange "first" and "second" as identifiers with the second image. The exchange of identifiers is performed simultaneously. After exchanging the identifiers, the structures can still be distinguished. The identifiers can be deleted. The structures after deleting the identifiers are distinguished by the figure marks. The order of the structures should not be interpreted only based on the records of the identifiers such as "first" and "second" in the present invention, and they should not be used as the basis for the existence of identifiers with small numbers.

[0114] Description of Reference Numerals

[0115] 1 Generation system

[0116] 10 Distance measuring device

[0117] 20. Filming Device

[0118] 30 Information processing device

[0119] 31 Ministry of Communications

[0120] 32 Output

[0121] 33 Storage

[0122] 34 Control Department

[0123] 40 First Image

[0124] 41 Second Image

[0125] 42, 48 Template Image

[0126] 43 Partial Image

[0127] 44 Range

[0128] Areas 45, 46, 47

Claims

1. An information processing device, comprising: The control unit acquires data of a first image and a second image generated by the camera in different postures, The control unit obtains the change in posture of the ranging device linked to the posture of the shooting device based on the change in posture of the shooting device, and the change in posture of the shooting device is calculated based on the position of the same subject in the first image and the position in the second image.

2. The information processing device according to claim 1, wherein: The control unit acquires position data indicating the position of an object existing around the distance measuring device. The control unit calculates a change in the posture of the photographing device based on the position of the subject in the first image and the position of the subject in the second image, which is within a range exceeding a specified distance from the photographing device, and the specified distance is a distance at which the ranging device cannot obtain the position data.

3. The information processing device according to claim 2, wherein: The control unit acquires the amount of change if the correlation between the first image and the second image of the image of the subject in the range exceeding the predetermined distance exceeds a predetermined value.

4. The information processing device according to any one of claims 1 to 3, wherein: The control unit acquires first data and second data from the distance measuring device, the first data being position data generated by the photographing device in a posture for generating the first image, and the second data being position data generated by the photographing device in a posture for generating the second image, The control unit adjusts a positional relationship between a position of the object represented by the first data and a position of the object represented by the second data according to the change amount.

5. The information processing device according to claim 4, wherein: The control unit generates an environment map based on the positions of the objects after the positional relationship is adjusted.

6. The information processing device according to claim 3, wherein: The control unit sets the size of the image of the subject according to the speed at which the posture of the distance measuring device changes.

7. The information processing device according to claim 6, wherein: The control unit sets the image so that the size of the image of the subject becomes smaller as the speed at which the posture of the distance measuring device changes becomes faster.

8. The information processing device according to claim 1, wherein: The control unit performs template matching processing on the first image and the second image to obtain a difference between a position of the same subject in the first image and a position of the same subject in the second image. In the template matching process, an image portion similar to a portion of the first image set as a template image is detected from the second image.

9. An information processing method, comprising: Acquire data of a first image and data of a second image generated by a camera in different postures, According to the change in the posture of the shooting device, the change in the posture of the ranging device linked to the posture of the shooting device is obtained, and the change in the posture of the shooting device is calculated based on the position of the same subject in the first image and the position in the second image.

10. A procedure, Cause a computer to perform an action comprising the following steps: Acquire data of a first image and data of a second image generated by a camera in different postures, According to the change in the posture of the shooting device, the change in the posture of the ranging device linked to the posture of the shooting device is obtained, and the change in the posture of the shooting device is calculated based on the position of the same subject in the first image and the position in the second image.

Images