Information processing method, information processing device, and program

Abstract

An information processing method according to an aspect of the present disclosure: acquires (S110) image information indicating an image generated by an imaging device; performs first conversion (S130) of an image using a conversion table for correcting image distortion, which is dependent on a lens included in the imaging device, and image conversion based on a change in the roll angle of the imaging device; and performs second conversion (S140) for further converting the image subjected to the first conversion, on the basis of prescribed information.

Description

Information processing method, information processing device, and program

[0001] The present disclosure relates to an information processing method, an information processing device, and a program.

[0002] 2. Description of the Related Art Conventionally, there is a technique for correcting an image in accordance with the performance of a lens included in an imaging device (see, for example, Japanese Patent Application Laid-Open No. 2003-124998).

[0003] Patent Document 1 discloses an apparatus equipped with an image correction processing unit that corrects distortion in images captured by a fisheye lens camera. The image correction processing unit performs calculations by combining coordinate transformations that correct the installation angle of the fisheye lens camera and coordinate transformations that correct distortion in the fisheye lens image.

[0004] Patent No. 4048511

[0005] It is desirable to reduce the amount of processing required in transforming images obtained from an imaging device, such as by correcting the images.

[0006] The present disclosure provides an information processing method and the like that can reduce the amount of image conversion processing.

[0007] An information processing method according to one aspect of the present disclosure acquires image information indicating an image generated by an imaging device, performs a first transformation to transform the image using a transformation table for correcting distortion of the image that depends on a lens possessed by the imaging device and for transforming the image based on changes in the roll angle of the imaging device, and performs a second transformation to further transform the image after the first transformation based on specified information.

[0008] An information processing device according to one aspect of the present disclosure includes an acquisition unit that acquires an image generated by an imaging device, and (i) a conversion unit that performs a first conversion to convert the image using a conversion table for correcting distortion of the image that depends on a lens possessed by the imaging device and converting the image based on a change in the roll angle of the imaging device, and (ii) a conversion unit that performs a second conversion to further convert the image after the first conversion based on predetermined information.

[0009] A program according to one aspect of the present disclosure is a program for causing a computer to execute the information processing method.

[0010] According to the information processing method and the like according to one aspect of the present disclosure, the amount of processing required for image conversion can be reduced.

[0011] Fig. 1 is a block diagram showing a configuration of an information processing system according to an embodiment. Fig. 2 is a diagram for explaining a specific example of an image according to an embodiment. Fig. 3 is a diagram for explaining the attitude of an imaging device according to an embodiment. Fig. 4 is a diagram for explaining correction of image distortion dependent on a lens included in an imaging device according to an embodiment. Fig. 5 is a diagram for explaining image conversion based on a change in the roll angle of an imaging device according to an embodiment. Fig. 6 is a flowchart showing an information processing method according to an embodiment.

[0012] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0013] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims of the present disclosure are described as optional components.

[0014] In this specification and drawings, the X-axis, Y-axis, and Z-axis represent the three axes of a three-dimensional Cartesian coordinate system. Similarly, in this specification and drawings, the X1-axis, Y1-axis, and Z1-axis represent the three axes of a three-dimensional Cartesian coordinate system. Similarly, in this specification and drawings, the X2-axis, Y2-axis, and Z2-axis represent the three axes of a three-dimensional Cartesian coordinate system.

[0015] The X, Y, and Z axes represent a world coordinate system, the X1, Y1, and Z1 axes represent a coordinate system based on the imaging device (also called a camera coordinate system), and the X2, Y2, and Z2 axes represent a coordinate system in the image (also called an image coordinate system).

[0016] Furthermore, unless otherwise specified, ordinal numbers such as "first" and "second" do not refer to the number or order of components, but are used to avoid confusion and distinguish between components of the same type.

[0017] (Embodiment) [Configuration] FIG. 1 is a block diagram showing the configuration of an information processing system 10 according to this embodiment.

[0018] The information processing system 10 is a system that converts images. Specifically, the information processing system 10 converts an image obtained from the imaging device 200, which captures an image of a subject (object) such as a person, based on information about the imaging device 200. The imaging device 200 is disposed in, for example, a vehicle and captures an image of the vehicle's surroundings. When the position and / or orientation (hereinafter also simply referred to as the position and orientation) at which the imaging device 200 captures the subject or the like is changed (for example, when it is assumed that the position and / or orientation has changed), the information processing system 10 converts the image captured by the imaging device 200 (specifically, an image generated by the imaging device 200 capturing an image of the subject or the like) in accordance with the change.

[0019] The information processing system 10 includes an information processing device 100 , an imaging device 200 , and a display device 300 .

[0020] The information processing device 100 is a computer that converts images. The information processing device 100 is realized, for example, by a communication interface for communicating with the imaging device 200, a non-volatile memory in which a program is stored, a volatile memory that is a temporary storage area for executing the program, an input / output port for transmitting and receiving signals, and a processor that executes the program. The communication interface may be realized, for example, by an antenna and a wireless communication circuit to enable wireless communication, or by a connector to which a communication line is connected to enable wired communication. The information processing device 100 is mounted, for example, on a vehicle, but may also be located outside the vehicle.

[0021] The imaging device 200 is a camera that captures (generates) an image to be converted by the information processing device 100 .

[0022] FIG. 2 is a diagram for explaining a specific example of an image according to the embodiment.

[0023] Assume that the imaging device 200 captures an image in the positive direction of the Z axis, for example, as shown in (a) of FIG. 2 . In this case, assume that the imaging device 200 generates, for example, an image (also referred to as a first image) as shown in (b) of FIG. 2 . In this case, assume that the information processing device 100 changes the imaging direction of the imaging device 200 downward along the Y axis, for example, as shown in (c) of FIG. 2 , and moves (translates) the position of the imaging device 200 above the subject, as shown in (d) of FIG. 2 , while maintaining the orientation shown in (c) of FIG. 2 . The information processing device 100 generates, for example, an image (also referred to as a second image) that is estimated to be generated by the imaging device 200 when imaging is performed at such a position and orientation of the imaging device 200, by transforming the first image based on the amount of change in the position and orientation of the imaging device 200.

[0024] The imaging device 200 is disposed in a vehicle, for example, and generates an image by capturing an image of the surroundings, such as the front, rear, or sides of the vehicle.

[0025] The vehicle may be, for example, a motorcycle or an automobile, but may also be a mobile robot such as an AMR (Autonomous Mobile Robot). The vehicle may be operated by a driver on board the vehicle, may be remotely controlled, or may be capable of autonomous driving. The imaging device 200 may also be disposed outside the vehicle. The imaging device 200 may also be held by a user or the like.

[0026] Furthermore, the subject captured by the imaging device 200 may be, for example, a person, an object, a landscape, or any other target.

[0027] The imaging device 200 generates an image (image information) including color information indicating the color of the subject, such as an RGB (Red, Green, Blue) image.

[0028] Note that the imaging device 200 may be, for example, a camera that generates an image including color information indicating the color of the subject and distance information indicating the distance between the imaging device 200 and the subject. That is, the image generated by the imaging device 200 may be an image including color information indicating the color of the subject and distance information indicating the distance between the imaging device 200 and the subject. That is, the image information may include color information indicating the color of each pixel in the image and distance information indicating the distance between the imaging device 200 and a position corresponding to each pixel. In other words, the image may be an RGB-D (depth) image.

[0029] The imaging device 200 includes a solid-state imaging element that detects light, such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), a lens such as a fisheye lens that guides incident light to the solid-state imaging element, and a housing that houses the solid-state imaging element and the lens. The imaging device 200 may also include a distance measurement sensor such as a LiDAR (Light Detection and Ranging). The types of the solid-state imaging element, lens, and distance measurement sensor included in the imaging device 200 may each be determined arbitrarily and are not particularly limited.

[0030] The display device 300 is a display that displays images. Specifically, the display device 300 displays images generated by the imaging device 200, images converted by the information processing device 100, and the like.

[0031] Next, the configuration of the information processing device 100 will be specifically described.

[0032] The information processing device 100 includes an acquisition unit 110 , a conversion unit 120 , an output unit 130 , and a storage unit 140 .

[0033] The acquisition unit 110 is a processing unit that acquires image information (first image information) indicating an image generated by the imaging device 200. For example, the acquisition unit 110 directly acquires image information from the imaging device 200 that is communicably connected to the information processing device 100 by communicating with the imaging device 200. Note that, when an image is stored in the storage unit 140, the acquisition unit 110 may acquire the image information from the storage unit 140.

[0034] The acquisition unit 110 also acquires movement information indicating the amount of change in the position and orientation of the image capture device 200 .

[0035] The movement information is information that indicates the amount of change in the position and orientation of the image capture device 200. Specifically, the movement information indicates the amount of change in the pitch angle, yaw angle, and roll angle of the image capture device 200 from a predetermined orientation, and the amount of change in translation (shift movement) of the image capture device 200 from a predetermined position.

[0036] The information indicating the amount of change may be information indicating a change from a predetermined position and orientation of the imaging device 200 (for example, the current position and orientation when the imaging device 200 performed imaging), and may be information indicating the amount of change in angle and / or position (distance) from the predetermined position and orientation, or information indicating the angle and / or position after the change from the predetermined position and orientation.

[0037] 3 is a diagram for explaining the attitude of the imaging device 200 according to the embodiment. Specifically, FIG. 3 shows the relationship between the camera coordinate system and the pitch angle, yaw angle, and roll angle in the present embodiment.

[0038] The camera coordinate system is a coordinate system that uses the position of the image capture device 200 as a reference (i.e., an origin). Note that the position of the origin in the camera coordinate system may be any position on the image capture device 200.

[0039] In this embodiment, the imaging direction of the imaging device 200 is indicated by the Z1 axis direction. Specifically, the direction in which the imaging device 200 captures images is indicated by the positive direction of the Z1 axis. The rotation angle (rotation angle) around the Z1 axis is referred to as the roll angle. The axes that are perpendicular to the Z1 axis and extend in directions perpendicular to each other are referred to as the X1 axis and the Y1 axis, respectively. The rotation angle around the X1 axis is referred to as the pitch angle. The rotation angle around the Y1 axis is referred to as the yaw angle.

[0040] The movement information is an example of predetermined information.

[0041] In addition, the movement information may include at least one of information indicating the amount of change in the pitch angle of the imaging device 200, information indicating the amount of change in the yaw angle of the imaging device 200, information indicating the amount of change in the roll angle of the imaging device 200, and information indicating the amount of change in the parallel movement of the imaging device 200 from a predetermined position.

[0042] The movement information may be stored in advance in the storage unit 140, or may be acquired from a user via a reception device such as a touch panel or a button. The movement information may also be acquired from various sensors, such as a gyro sensor and a GPS (Global Positioning System), provided in the image capture device 200 and / or a vehicle in which the image capture device 200 is installed. In other words, the movement information may be information indicating the pitch angle, yaw angle, roll angle, and amount of change (amount of movement) in position that the image capture device 200 has actually changed.

[0043] The conversion unit 120 is a processing unit that converts the image indicated by the image information acquired by the acquisition unit 110. Specifically, the conversion unit 120 performs a first conversion that converts the image using a conversion table.

[0044] The conversion table is a table (table information) for converting an image. Specifically, the conversion table is a table for correcting image distortion that depends on the lens of the image capture device 200 and converting an image based on a change in the roll angle of the image capture device 200.

[0045] Here, the change in the image accompanying the change in the position and orientation of the image capturing device 200 is calculated by, for example, affine transformation. For example, the change in the image accompanying the change in the position and orientation of the image capturing device 200 is defined by the following equation (1).

[0046]

[0047] Note that the above formula (1) indicates the change in the position of a pixel of an image when the position and orientation of the image capture device 200 in a predetermined position and orientation is changed in the order of rotation around a roll angle (roll angle movement), rotation around a yaw angle (yaw angle movement), rotation around a pitch angle (pitch angle movement), and translation. x, y, and z respectively indicate the X2 coordinate, Y2 coordinate, and Z2 coordinate of the pixel before transformation. x ｎｅｗ , y ｎｅｗ and z ｎｅｗ indicate the X2 coordinate, Y2 coordinate, and Z2 coordinate of the pixel after transformation, respectively. θx indicates the rotation angle of the imaging device 200 around the yaw angle. θy indicates the rotation angle of the imaging device 200 around the pitch angle. θz indicates the rotation angle of the imaging device 200 around the roll angle. The first matrix from the left (first matrix) of the first term on the right side of Equation (1) corresponds to the pitch angle movement of the imaging device 200. The second matrix from the left (second matrix) of the first term on the right side of Equation (1) corresponds to the yaw angle movement of the imaging device 200. The third matrix from the left (third matrix) of the first term on the right side of Equation (1) corresponds to the roll angle movement of the imaging device 200. The second term on the right side of Equation (1) (AX, AY, and AZ) corresponds to the translation amount of the imaging device 200 (the distance by which the imaging device 200 is translated).

[0048] In the above formula (1), when attention is focused only on the first matrix of the first term on the right side, the second matrix of the first term on the right side, the third matrix of the first term on the right side, and the third matrix of the first term on the right side of the second term on the right side, that is, the roll angle movement, the above formula (1) is transformed into the following formula (2).

[0049]

[0050] If the above formula (2) is transformed with a focus only on the roll angle movement, the z term in the first line on the right side becomes 0, and the x and y terms in the second and third lines on the right side become 0. In other words, the x value and the y value do not affect the z value. Similarly, the z value does not affect the x value and the y value.

[0051] Therefore, the roll angle movement only affects the movement of pixel position along the X2 axis and the Y2 axis, and does not affect the movement of pixel position along the Z2 axis (i.e., it can be ignored).

[0052] FIG. 4 is a diagram illustrating correction of image distortion dependent on a lens included in the imaging device 200 according to the embodiment. Note that in FIG. 4 and FIG. 5 (described later), a coordinate system (image coordinate system) for an image generated by the imaging device 200 is indicated by the X2, Y2, and Z2 axes. The image coordinate system in this embodiment is a coordinate system in which the origin is the position of a pixel at a predetermined position (e.g., the center of the image) of the image when the entire image is assumed to be displayed on a display device 300 such as a display, the rightward direction from that pixel is the positive X2 axis, and the upward direction from that pixel is the positive Y2 axis. In this embodiment, the direction of the X1 axis corresponds to the direction of the X2 axis, the direction of the Y1 axis corresponds to the direction of the Y2 axis, and the direction of the Z1 axis corresponds to the direction of the Z2 axis.

[0053] When a fisheye lens or the like is used in the imaging device 200, the image generated by the imaging device 200 is distorted by the influence of the lens used in the imaging device 200, causing the position of each pixel in the image to change, as shown in FIG. 4A, for example. In particular, the positions of pixels located on the outer edges of the image are distorted significantly. Therefore, conventionally, a correction table is used to convert (correct) the image, changing the position of each pixel in the image to a state where it is not affected by the lens, as shown in FIG. 4B, for example.

[0054] FIG. 5 is a diagram for explaining image conversion based on a change in the roll angle of the imaging device 200 according to the embodiment.

[0055] When imaging is performed by the imaging device 200 while the roll angle of the imaging device 200 is changed, in other words, when imaging is performed by the imaging device 200 while the imaging device 200 is rotated around the Z1 axis, an image rotated around the Z2 axis is generated by the imaging device, as shown in (a) of FIG. 5. In order to convert such an image into an image in which the roll angle of the imaging device 200 has not been changed, it is conceivable to rotate the image around the Z2 axis, as shown in equation (2) above. Therefore, in order to convert an image in accordance with changes in the roll angle of the imaging device 200, it is not necessary to convert each pixel in the Z2 axis direction, and it is conceivable that it is sufficient to convert the position of each pixel within the X2Y2 plane.

[0056] From the above, it can be considered that both correction of lens-dependent image distortion and conversion of an image in accordance with changes in the roll angle of the imaging device 200 can be achieved by converting the position of each pixel within the X2Y2 plane.

[0057] Therefore, in this embodiment, the conversion unit 120 not only corrects the lens-dependent image distortion using a conversion table, but also performs image conversion based on changes in the roll angle of the imaging device 200 all at once.

[0058] Specifically, for example, the converter 120 generates a conversion table based on the amount of change in roll angle included in the movement information and a correction table that corrects image distortion that depends on the lens of the imaging device 200. Specifically, the converter 120 generates the conversion table by calculating the product of the correction table that corrects image distortion that depends on the lens of the imaging device 200 and a matrix corresponding to the roll angle movement (specifically, the third matrix described above). More specifically, the converter 120 substitutes the amount of change in roll angle (roll angle from a predetermined position) included in the movement information into the third matrix (specifically, θz in the third matrix), and calculates the product of the third matrix into which the amount of change in roll angle has been substituted and the correction table that corrects image distortion that depends on the lens of the imaging device 200, thereby generating the conversion table. The generated conversion table is stored in, for example, the storage unit 140.

[0059] The correction table may be stored in advance in storage unit 140. The conversion table may not be calculated by converter 120 but may be stored in advance in storage unit 140. Information indicating the product of the correction table and the third matrix may be stored in advance in storage unit 140. In this case, for example, when information indicating the amount of change in the roll angle is acquired, converter 120 may calculate (generate) the conversion table based on the information indicating the product of the correction table and the third matrix and the information indicating the amount of change in the roll angle.

[0060] Furthermore, the conversion unit 120 performs a second conversion based on predetermined information to further convert the image that has undergone the first conversion. In the second conversion, the image is converted using, for example, a predetermined mathematical formula. That is, the conversion unit 120 converts the image (specifically, the position of each pixel in the image) using a conversion table, and then further converts the converted image (specifically, the position of each pixel in the converted image) using a mathematical formula.

[0061] The conversion unit 120 further converts the image converted using the conversion table, for example, using the following equation (3).

[0062]

[0063] The above equation (3) is an equation obtained by removing the third matrix from the above equation (1). That is, in the second transformation, processing corresponding to the roll angle movement is not performed.

[0064] The predetermined information includes, for example, information indicating the amount of change in the pitch angle of the imaging device 200. The predetermined information also includes, for example, information indicating the amount of change in the yaw angle of the imaging device 200. The predetermined information also includes, for example, information indicating the amount of change in the position of the imaging device 200. In this way, the predetermined information includes, for example, information indicating the amount of change in the position and orientation of the imaging device 200. In this embodiment, the predetermined information is information included in the movement information.

[0065] The output unit 130 is a processing unit that outputs image information (second image information) indicating the image (second image) that has been subjected to the second conversion. The output unit 130 outputs the image information to the display device 300, for example, to cause the display device 300 to display the second image.

[0066] Note that the output unit 130 may cause the display device 300 to display an image (first image) before the first conversion. For example, the output unit 130 may first display the first image on the display device 300, and then, upon receiving an instruction from the user via the reception device indicating that a second image is to be displayed, cause the output unit 130 to display the second image on the display device 300. In this case, the output unit 130 may cause the display device 300 to display only the second image out of the first image and the second image, or may cause the display device 300 to display both the first image and the second image. The instruction may also be movement information. For example, when the acquisition unit 110 acquires movement information as the instruction, the conversion unit 120 generates the second image by converting the first image based on the movement information, and the output unit 130 outputs image information indicating the second image to the output unit 130, thereby displaying the second image on the display device 300.

[0067] Each processing unit such as the acquisition unit 110, the conversion unit 120, and the output unit 130 is realized, for example, by one or more memories that store a control program and one or more processors such as a CPU (Central Processing Unit) that executes the control program.

[0068] The storage unit 140 is a storage device that stores various types of information. The storage unit 140 stores, for example, images converted by the information processing device 100, images generated by the imaging device 200, correction tables (correction table information), conversion tables (conversion table information), movement information, etc. The storage unit 140 is realized by, for example, a semiconductor memory or an HDD (Hard Disk Drive).

[0069] [Processing Procedure] Next, a processing procedure of the information processing device 100 according to the embodiment will be described.

[0070] 6 is a flowchart showing an information processing method according to an embodiment. For example, the information processing device 100 includes a processor and a memory, and the processor performs the following processes using the memory.

[0071] First, the information processing device 100 acquires image information indicating an image generated by the imaging device 200 (S110). Specifically, the acquisition unit 110 included in the information processing device 100 acquires first image information indicating a first image from the imaging device 200. For example, the information processing device 100 controls the imaging device 200 to cause the imaging device 200 to capture an image to generate an image, and acquires first image information indicating the first image generated thereby.

[0072] Next, the information processing device 100 acquires movement information (S120). The movement information may be acquired from the storage unit 140, from the user via the reception device, or from various sensors provided in the image capture device 200 and / or the vehicle in which the image capture device 200 is installed.

[0073] Next, the information processing device 100 converts the image using a conversion table (S130). Specifically, the information processing device 100 performs a first conversion to convert the first image using a conversion table for correcting image distortion dependent on the lens of the imaging device 200 and converting the image based on a change in the roll angle of the imaging device 200. The information processing device 100 generates the conversion table using, for example, the correction table stored in the storage unit 140, information indicating the amount of change in the roll angle included in the movement information, and the above formula (2). Furthermore, the information processing device 100 converts the first image using, for example, the generated conversion table.

[0074] Next, specifically, the information processing device 100 further transforms the image transformed in step S130 based on predetermined information (S140). Specifically, the information processing device 100 performs a second transformation, which further transforms the first image transformed in the first transformation, based on the predetermined information. The information processing device 100 generates a second image by transforming the position of each pixel in the first image using, for example, at least one of information indicating the amount of change in pitch angle, information indicating the amount of change in yaw angle, and information indicating the amount of change in position, which are included in the movement information, and the above-mentioned equation (3).

[0075] Next, the information processing device 100 outputs image information indicating the image further converted in step S140 (S150). For example, the output unit 130 included in the information processing device 100 outputs second image information indicating the second image generated by converting the first image in step S140 to the display device 300, thereby displaying the second image on the display device 300.

[0076] As described above, the information processing device 100, the imaging device 200, and the display device 300 are disposed in, for example, a vehicle. For example, when the vehicle starts to be driven, the information processing device 100 executes steps S110 to S150 to generate a conversion table and causes the display device 300 to display a second image using the generated conversion table. For example, the information processing device 100 converts an image generated by the imaging device 200 capturing an image of the front of the vehicle into an image of the front of the vehicle as viewed from above. As the vehicle travels, the position of the imaging device 200 changes depending on the vehicle's traveling speed. For example, the information processing device 100 repeatedly acquires information indicating the traveling speed from the vehicle, and repeatedly generates and displays a second image on the display device 300 using the acquired information, movement information, and the generated conversion table. With this usage method, the amount of change in the roll angle of the imaging device 200 is fixed, so once the conversion table is generated, it can be repeatedly used to generate the second image.

[0077] [Effects, etc.] As described above, the information processing device 100 according to this embodiment acquires an image generated by the imaging device 200, performs a first conversion to convert the image using a conversion table for correcting image distortion that depends on the lens of the imaging device 200 and converting the image based on changes in the roll angle of the imaging device 200, and then executes a second conversion to further convert the image that has undergone the first conversion based on predetermined information.

[0078] This allows image conversion based on changes in the roll angle of the image capture device 200 to be performed in a lump using a conversion table, along with correction of lens-dependent image distortion.

[0079] As described above, the information processing device 100 converts an image obtained from the imaging device 200, for example, so as to generate an image that is estimated to be generated when the imaging device 200 captures an image from another specific direction, such as from above the vehicle, based on an image generated by the imaging device 200 capturing an image in a specific direction, such as the front of the vehicle.

[0080] In this manner, when the information processing device 100 repeatedly generates images estimated to be generated when the imaging direction of the imaging device 200 is changed from a specific first direction to a specific second direction different from the specific first direction, the conversion table, once generated, can be repeatedly used to generate the second image, as long as the amount of change in the roll angle of the imaging device 200 does not change. Therefore, when converting such images, the information processing device 100 does not need to use the correction table and the above formula (1) each time, but can simply use the conversion table and the above formula (3). Therefore, the information processing device 100 can reduce the amount of processing required for image conversion. Specifically, the amount of calculation related to the third matrix can be reduced. Furthermore, when such calculations are implemented by a circuit, the total number of multipliers performing such calculations can be reduced. For example, when the above formula (1) is used, a total of 25 multipliers are required, whereas when the above formula (3) is used, only 12 multipliers are required. Therefore, the implementation area of ​​the circuit performing such calculations can be reduced. Furthermore, because the total number of multipliers is reduced, the power consumption of the circuit performing such calculations can be reduced.

[0081] For example, the specified information includes at least one of information indicating the amount of change in the pitch angle of the imaging device 200, information indicating the amount of change in the yaw angle of the imaging device 200, and information indicating the amount of change in the position of the imaging device 200.

[0082] This allows the amount of processing required for converting an image to be reduced even when at least one of the pitch angle, yaw angle, and position of the image capture device 200 changes each time the image is converted.

[0083] Furthermore, for example, the image information includes color information indicating the color of each pixel in the image, and distance information indicating the distance between the position corresponding to each pixel and the imaging device 200. In other words, the image acquired by the acquisition unit 110 is, for example, an RGB-D image.

[0084] This allows the conversion of RGB-D images based on changes in the roll angle of the image capture device 200 to be performed in one step using a conversion table, along with correction of lens-dependent image distortion, thereby reducing the amount of processing required for converting RGB-D images.

[0085] (Other Embodiments) As described above, the embodiments have been described as examples of the technology according to the present disclosure. However, the technology according to the present disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. For example, the following modifications are also included in one embodiment of the present disclosure.

[0086] For example, when the present disclosure is realized as a program (software), each step is performed by running the program using hardware resources such as a computer's CPU, memory, input / output circuits, etc. In other words, each step is performed by the CPU acquiring data from memory or input / output circuits, etc., performing calculations, and outputting the calculation results to memory or input / output circuits, etc.

[0087] Furthermore, the computer that executes this program may be a single computer or multiple computers, and may perform centralized processing or distributed processing.

[0088] In the above embodiment, each component included in information processing device 100 may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0089] Some or all of the functions of the information processing device 100 according to the above-described embodiment are typically realized as an LSI, which is an integrated circuit. These may be individually integrated into single chips, or some or all of them may be integrated into a single chip. Furthermore, the integrated circuit is not limited to an LSI, and may be realized using a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array), which can be programmed after LSI manufacturing, or a reconfigurable processor, which allows the connections and settings of circuit cells within an LSI to be reconfigured.

[0090] Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, it is natural that each component included in the information processing device 100 may be integrated using that technology.

[0091] Furthermore, for example, in the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit, the order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0092] In addition, this disclosure also includes forms obtained by making various modifications to the embodiments that a person skilled in the art would think of, and forms realized by arbitrarily combining the components and functions in each embodiment within the scope that does not deviate from the intent of this disclosure.

[0093] (Additional Notes) The above description of the embodiments discloses the following techniques.

[0094] (Technology 1) An information processing method that acquires image information indicating an image generated by an imaging device, performs a first transformation to transform the image using a transformation table for correcting distortion of the image that depends on a lens possessed by the imaging device and for transforming the image based on changes in the roll angle of the imaging device, and performs a second transformation to further transform the image that has undergone the first transformation based on specified information.

[0095] This allows image conversion based on changes in the roll angle of the imaging device to be performed collectively using a conversion table, along with correction of lens-dependent image distortion, thereby reducing the amount of processing required for image conversion.

[0096] (Technology 2) The information processing method according to Technology 1, wherein the predetermined information includes information indicating a change in pitch angle of the imaging device.

[0097] With this, even when the pitch angle of the imaging device is changed, the image can be converted in accordance with the amount of change in the pitch angle.

[0098] (Technology 3) The information processing method according to Technology 1 or 2, wherein the predetermined information includes information indicating a change in yaw angle of the imaging device.

[0099] This allows the image to be converted in accordance with the amount of change in the pitch angle even when the yaw angle of the imaging device is changed.

[0100] (Technology 4) The information processing method according to any one of techniques 1 to 3, wherein the predetermined information includes information indicating an amount of change in the position of the imaging device.

[0101] This allows the image to be converted in accordance with the amount of change in position even when the position of the imaging device is changed.

[0102] (Technology 5) An information processing method according to any one of techniques 1 to 4, wherein the image information includes color information indicating the color of each pixel in the image, and distance information indicating the distance between the corresponding position of each pixel and the imaging device.

[0103] In other words, the information processing method according to one aspect of the present disclosure is applicable not only to two-dimensional images such as RGB images, but also to RGB-D images. This allows for image conversion based on changes in the roll angle of the imaging device, along with correction of lens-dependent image distortion, all at once using a conversion table. This reduces the amount of processing required for RGB-D image conversion.

[0104] (Technology 6) An information processing device comprising: an acquisition unit that acquires image information indicating an image generated by an imaging device; and a conversion unit that (i) performs a first conversion to convert the image using a conversion table for correcting distortion of the image that depends on a lens of the imaging device and converting the image based on a change in the roll angle of the imaging device; and (ii) performs a second conversion to further convert the image that has undergone the first conversion based on predetermined information.

[0105] This provides the same effect as the information processing method described in Technology 1.

[0106] (Technology 7) A program for causing a computer to execute the information processing method according to any one of technologies 1 to 5.

[0107] This provides the same effect as the information processing method described in Technology 1.

[0108] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0109] The present disclosure can be applied to devices that convert images, etc.

[0110] REFERENCE SIGNS LIST 10 Information processing system 100 Information processing device 110 Acquisition unit 120 Conversion unit 130 Output unit 140 Storage unit 200 Imaging device 300 Display device

Claims

1. An information processing method comprising: acquiring image information indicating an image generated by an imaging device; performing a first transformation to transform the image using a transformation table for correcting distortion of the image that depends on the lens of the imaging device and for transforming the image based on changes in the roll angle of the imaging device; and performing a second transformation to further transform the image that has undergone the first transformation based on specified information.

2. The information processing method according to claim 1, wherein the predetermined information includes information indicating a change in pitch angle of the imaging device.

3. The information processing method according to claim 1, wherein the predetermined information includes information indicating the amount of change in the yaw angle of the imaging device.

4. The information processing method according to claim 1, wherein the predetermined information includes information indicating the amount of change in the position of the imaging device.

5. The information processing method according to claim 1, wherein the image information includes color information indicating the color of each pixel in the image, and distance information indicating the distance between the position corresponding to each pixel and the imaging device.

6. An information processing device comprising: an acquisition unit that acquires an image generated by an imaging device; and (i) a conversion unit that performs a first conversion to convert the image using a conversion table for correcting distortion of the image that depends on the lens of the imaging device and converting the image based on changes in the roll angle of the imaging device; and (ii) a conversion unit that performs a second conversion to further convert the image that has undergone the first conversion based on specified information.

7. A program for causing a computer to execute the information processing method according to any one of claims 1 to 5.

Images