Image processing device, information processing device, image processing method, and program

Abstract

To suitably perform tilt correction with respect to a camera image.SOLUTION: An image processing device comprises: a first acquisition unit which acquires a camera image captured by a camera attached to a gimbal; a second acquisition unit which refers to at least one of the camera image and gimbal information including a tilt of the gimbal and acquires a tilt angle of the camera; and a tilt correction unit which generates an image after the tilt correction by applying tilt correction processing which refers to the tilt angle of the camera to the camera image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an information processing device, an image processing method, and a program. [Background technology]

[0002] There are known techniques for acquiring images captured by a camera attached to a gimbal and using them for various purposes. For example, Patent Document 1 discloses a technique for using data captured by a camera attached to a drone via a gimbal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO2021-256464 publication Summary of the Invention [Problem to be solved by the invention]

[0004] Using a gimbal is expected to keep the camera attitude as horizontal as possible, but in reality, even if a gimbal is used, it is not always possible to keep the camera attitude horizontal because there is a limit to the angle at which the gimbal can maintain horizontality. For this reason, the technology described in Patent Document 1 has an issue with how to handle cases where the camera attached to the gimbal is tilted.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and one exemplary purpose thereof is to provide a technology that can suitably perform tilt correction on camera images captured by a camera attached to a gimbal. [Means for solving the problem]

[0006] An image processing device according to an exemplary aspect of the present disclosure includes a first acquisition means for acquiring a camera image captured by a camera attached to a gimbal, a second acquisition means for acquiring a tilt angle of the camera by referring to at least one of the camera image and gimbal information including the tilt of the gimbal, and a tilt correction means for applying a tilt correction process to the camera image with reference to the tilt angle of the camera to generate a tilt-corrected image.

[0007] An information processing device according to an exemplary aspect of the present disclosure includes the image processing device described above, an interpolation means for applying an interpolation process to the tilt-corrected image to generate an interpolated image, an image recognition means for applying an image recognition process to the interpolated image, and an output means for outputting the results of the image recognition process.

[0008] An information processing device according to an exemplary aspect of the present disclosure includes the image processing device described above, an image recognition unit that applies image recognition processing to the tilt-corrected image, and an output unit that outputs the results of the image recognition processing.

[0009] An information processing method according to an exemplary aspect of the present disclosure includes acquiring a camera image captured by a camera attached to a gimbal, acquiring a tilt angle of the camera by referring to at least one of the camera image and gimbal information including the tilt of the gimbal, and applying a tilt correction process to the camera image that refers to the tilt angle of the camera, thereby generating a tilt-corrected image.

[0010] In addition, the information processing device according to each aspect may be realized by a computer. In this case, a program for realizing the information processing device on a computer by causing the computer to operate as each means provided in the information processing device, and a computer-readable recording medium on which the program is recorded, also fall within the scope of the present invention. [Effects of the Invention]

[0011] According to one exemplary aspect of the present disclosure, it is possible to provide an exemplary effect of suitably correcting tilt of a camera image captured by a camera attached to a gimbal. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram illustrating a configuration of an image processing device according to the present disclosure. [Figure 2] 1 is a flowchart showing the flow of an image processing method according to the present disclosure. [Figure 3] 1 is a block diagram illustrating a configuration of an information processing system according to the present disclosure. [Figure 4] FIG. 10 is a diagram for explaining a processing example performed by the information processing system according to the present disclosure. [Figure 5] FIG. 10 is a diagram for explaining a processing example performed by the information processing system according to the present disclosure. [Figure 6] FIG. 10 is a diagram for explaining a processing example performed by the information processing system according to the present disclosure. [Figure 7] FIG. 10 is a diagram for explaining a processing example performed by the information processing system according to the present disclosure. [Figure 8] FIG. 10 is a diagram for explaining a processing example performed by the information processing system according to the present disclosure. [Figure 9] FIG. 10 is a diagram for explaining an example of output by the information processing system according to the present disclosure. [Figure 10] FIG. 1 is a block diagram illustrating a configuration of a computer that functions as an information processing device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0014] First Exemplary Embodiment A first exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form of each exemplary embodiment described later. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referred to in describing this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise.

[0015] (Configuration of image processing device 1) The configuration of an image processing device 1 according to this exemplary embodiment will be described below with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the image processing device 1 according to this exemplary embodiment. As shown in Fig. 1, the image processing device 1 includes a first acquisition unit 11, a second acquisition unit 12, and a tilt correction unit 13.

[0016] (First acquisition unit 11) The first acquisition unit 11 acquires a camera image. Here, the camera image is, for example, an image captured by a camera attached to a gimbal. As an example, the first acquisition unit 11 acquires the camera image from the camera via a wired or wireless communication path. Alternatively, the camera image from the camera may be temporarily stored in a storage unit, and the first acquisition unit 11 may acquire the camera image from the storage unit. Note that the term "gimbal" is merely an example of a name for a mechanism for stabilizing a camera, and this term does not limit the present exemplary embodiment. A gimbal may also be expressed as, for example, a "stabilizer" or an "electric stabilizer," and aspects using these terms are also included in the present exemplary embodiment.

[0017] (Second acquisition unit 12) The second acquisition unit 12 acquires the tilt angle of the camera by referring to at least one of the camera image and gimbal information including the tilt of the gimbal. As an example, the second acquisition unit 12 may be configured to derive the tilt angle of the camera by referring to the camera image.

[0018] The second acquisition unit 12 may be configured to derive the tilt angle of the camera with reference to the tilt of the gimbal. The second acquisition unit 12 may be configured to acquire the tilt angle of the camera with further reference to sensing data from an acceleration sensor or a gyro sensor attached to the camera or the gimbal.

[0019] (Tilt correction unit 13) The tilt correction unit 13 generates a tilt-corrected image by applying tilt correction processing, which refers to the camera tilt angle acquired (derived) by the second acquisition unit 12, to the camera image acquired by the first acquisition unit 11. The tilt-corrected image may be presented to the user via a display unit (not shown), for example, or may be input to an image recognition unit (not shown).

[0020] (Effects of image processing device 1) As described above, in the image processing device 1 according to this exemplary embodiment, Acquires camera images captured by a camera attached to a gimbal, acquiring a tilt angle of the camera by referring to at least one of the camera image and gimbal information including the tilt of the gimbal; A tilt correction process is applied to the camera image, with reference to the tilt angle of the camera, to generate a tilt-corrected image. According to the above configuration, The tilt that may be contained in a camera image captured by a camera attached to a gimbal is corrected by referring to at least one of the camera image and the gimbal information, so that tilt correction can be performed appropriately on the camera image.

[0021] (Flow of image processing method S1) Next, the flow of image processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of image processing method S1. As shown in Fig. 2, image processing method S1 includes a process (step, process) S11 for acquiring a camera image, a process (step, process) S12 for acquiring the tilt of the camera, and a process (step, process) S13 for generating an image after tilt correction.

[0022] (Step S11) In step S11, the first acquisition unit 11 acquires a camera image. Here, the camera image is, for example, an image captured by a camera attached to a gimbal. Specific processing by the first acquisition unit 11 has been described above, and therefore will not be described here.

[0023] (Step S12) Subsequently, in step S12, the second acquisition unit 12 acquires the tilt angle of the camera by referring to at least one of the camera image and gimbal information including the tilt of the gimbal. The specific processing by the second acquisition unit 12 has been described above, and therefore will not be described here.

[0024] (Step S13) Next, in step S13, the tilt correction unit 13 applies a tilt correction process that references the camera tilt angle acquired (derived) by the second acquisition unit 12 to the camera image acquired by the first acquisition unit 11, thereby generating a tilt-corrected image.

[0025] (Effect of image processing method S1) As described above, the image processing method S1 according to this exemplary embodiment: Acquires camera images captured by a camera attached to a gimbal, acquiring a tilt angle of the camera by referring to at least one of the camera image and gimbal information including the tilt of the gimbal; A tilt correction process is applied to the camera image, with reference to the tilt angle of the camera, to generate a tilt-corrected image. The above configuration provides the same effects as the image processing device 1.

[0026] Second Exemplary Embodiment A second exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0027] (Configuration of information processing system 100A) The configuration of an information processing system 100A according to this exemplary embodiment will be described with reference to FIG. 5. FIG. 3 is a block diagram showing the configuration of the information processing system 100A. As shown in FIG. 3, the information processing system 100A includes an information processing device 1A and a robot 50 connected to the information processing device 1A via a network N. The specific configuration of the network N does not limit this exemplary embodiment, and examples include a wireless local area network (LAN), a wired LAN, a wide area network (WAN), a public line network, a mobile data communication network, or a combination of these networks. It is not essential that the information processing system 100A includes the robot 50. The information processing device 1A may also be configured to acquire camera images from a camera attached to a gimbal provided on a device other than a robot (e.g., a drone or a self-propelled guided vehicle) or a gimbal supported by a person. In the following description, the term "information processing device 1A" does not limit this exemplary embodiment, and may also be referred to as an image processing device 1A, etc.

[0028] (Configuration of robot 50) In the example shown in FIG. 3, the robot 50 includes a robot body 51. As shown in FIG. 3, an arm 53 is attached to the robot body 51 via a joint 52. A gimbal 54 is attached to the tip of the arm 53, and a camera 55 is attached to the gimbal 54. As shown in FIG. 3, a sensor 56 may be attached directly or indirectly to the camera 55. The sensor 56 is, for example, an acceleration sensor or a gyro sensor, and is configured to be able to detect the tilt of the camera 55. The sensor 56 may be attached to the gimbal 54. However, the inclusion of the sensor 56 does not limit this exemplary embodiment, and a configuration without the sensor 56 is also included in this exemplary embodiment.

[0029] 3, the robot body 51 includes a control unit 511. Here, the control unit 511 includes, as an example, a control unit (not shown) that controls each part of the robot 50. Also, as shown in FIG. 3, the control unit 511 includes: Arm joint information AJI including at least one of the orientation of the arm 53 and the rotation angle of the joint 52 is acquired from a control unit, arm 53, or joint 52 (not shown), and is provided to the information processing device 1A via a communication unit (not shown). Gimbal information GI including tilt of gimbal 54 is acquired from a control unit, an arm 53, a joint 52, or a gimbal 54 (not shown), and is provided to the information processing device 1A via a communication unit (not shown). Image data (IMG) captured by the camera 55 is acquired from the camera 55 and provided to the information processing device 1A via a communication unit (not shown). The imaging data IMG may be, for example, a still image consisting of one frame, or a moving image consisting of multiple frames. Furthermore, when the imaging data IMG is a moving image, the images constituting each frame may be referred to as a camera image.

[0030] Note that the arm joint information AJI and the gimbal information GI may be configured so that one includes the other, or one replaces the other. For example, the tilt of the gimbal 54 may be derived from the arm joint information AJI by the control unit 511 or the control unit 20A. In such a configuration, the arm joint information AJI may replace the gimbal information GI. Furthermore, the gimbal information GI may be configured to include information indicating a limit value of the relative angle between the camera 55 and the gimbal 54 (also referred to as maximum angle information).

[0031] (Configuration of information processing device 1A) Next, the configuration of the information processing device 1A according to this exemplary embodiment will be described with reference to Fig. 3. As shown in Fig. 3, the information processing device 1A includes a control unit 20A, a storage unit 10A, a communication unit 30, and an input / output unit 40.

[0032] (Communication unit 30) The communication unit 30 communicates with devices external to the information processing device 1A. As an example, the communication unit 30 communicates with the robot 50. The communication unit 30 transmits data supplied from the control unit 20A to the robot 50, and supplies data received from the robot 50 to the control unit 20A.

[0033] The data acquired by the communication unit 30 from the robot 50 and provided to the control unit 20A includes the following: Arm joint information AJI Gimbal Information GI Image data (IMG) The arm control information AJI, gimbal information GI, and imaging data IMG have been described above, so a description thereof will be omitted here.

[0034] (Input / output section 40) The input / output unit 40 is configured to include at least one of input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel. Alternatively, the input / output unit 40 may be configured to have input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel connected to it. In this configuration, the input / output unit 40 accepts various types of information input to the information processing device 1A from the connected input devices. Furthermore, the input / output unit 40 outputs various types of information to the connected output devices under the control of the control unit 20A. An example of the input / output unit 40 is an interface such as a USB (Universal Serial Bus).

[0035] (Storage unit 10A) The storage unit 10A stores various data referenced by the control unit 20A and various data generated by the control unit 20A. Image data (IMG) Gimbal Information GI Tilt-corrected image TCI Output information OUT is stored. Here, the imaging data IMG has been described above, so a description thereof will be omitted. As described above, the gimbal information GI includes the tilt of the gimbal 54. Here, the information indicating the tilt is also referred to as gimbal tilt information GTI, as shown in FIG. 3. Furthermore, as described above, the gimbal information GI may include information indicating the limit value of the relative angle between the camera 55 and the gimbal 54 (maximum angle information MAI).

[0036] The tilt-corrected image TCI refers to, for example, an image whose tilt has been corrected by a tilt correction unit 13 (described later). Also, the output information OUT refers to, for example, information for output generated by an output unit 16 (described later). Specific examples of the tilt-corrected image TCI and the output information OUT will be described later.

[0037] (Control unit 20A) As shown in FIG. 3, the control unit 20A includes a first acquisition unit 11, a second acquisition unit 12, a tilt correction unit 13, an interpolation unit 14, an image recognition unit 15, and an output unit 16.

[0038] (First acquisition unit 11) The first acquisition unit 11 acquires a camera image (image data IMG) in the same manner as in the first exemplary embodiment. Here, as described above, the camera image (image data IMG) is, for example, an image captured by a camera 55 attached to a gimbal 54. Note that in this exemplary embodiment, the term "gimbal" is merely an example of the name of a mechanism for stabilizing a camera, and this term does not limit this exemplary embodiment. A gimbal may also be expressed as, for example, a "stabilizer" or an "electric stabilizer," and aspects using these terms are also included in this exemplary embodiment.

[0039] (Second acquisition unit 12) Similar to the exemplary embodiment 1, the second acquisition unit 12 acquires the tilt angle of the camera 55 by referring to at least one of the camera image (imaging data IMG) and gimbal information GI including the tilt of the gimbal 54. As an example, the second acquisition unit 12 may be configured to derive the tilt angle of the camera 55 by referring to the camera image (imaging data IMG).

[0040] The second acquisition unit 12 may be configured to derive the tilt angle of the camera 55 with reference to the tilt of the gimbal 54. As an example, the second acquisition unit 12 may Maximum angle information MAI indicating the limit value of the relative angle between the camera 55 and the gimbal 54; Gimbal 54 tilt and The second acquisition unit 12 may be configured to derive the tilt angle of the camera 55 by referring to the sensing data from the sensor 56 (acceleration sensor or gyro sensor) attached to the camera 55 or the gimbal 54. The specific processing by the second acquisition unit 12 will be described later.

[0041] (Tilt correction unit 13) As in the first exemplary embodiment, the tilt correction unit 13 generates a tilt-corrected image TCI by applying tilt correction processing that refers to the camera tilt angle acquired (derived) by the second acquisition unit 12 to the camera image (imaging data IMG) acquired by the first acquisition unit 11. For example, the tilt-corrected image TCI may be configured to be visually presented to the user via the input / output unit 40, or may be configured to be input to the image recognition unit 15 included in the control unit 20A. Specific processing by the tilt correction unit 13 will be described later.

[0042] (Interpolation unit 14) The interpolation unit 14 generates an interpolated image by applying interpolation processing to the tilt-corrected image TCI generated by the tilt correction unit 13. Specific processing by the interpolation unit 14 will be described later.

[0043] (Image Recognition Unit 15) The image recognition unit 15 The tilt-corrected image TCI generated by the tilt correction unit 13, and The interpolated image generated by the interpolation unit 14 Image recognition processing is applied to at least one of the above. Here, the image recognition processing may include, as an example, an object detection processing, but this is not intended to limit the present exemplary embodiment. Furthermore, as an example, the image recognition processing may be performed using an image recognition model that has been trained with reference to a group of images that include a mask region corresponding to an out-of-field region. Specific processing by the image recognition unit 15 will be described later.

[0044] (Output section 16) The output unit 16 presents the information derived by the above-described units included in the control unit 20A to the user via the input / output unit 40, or provides it to another device via the communication unit 30. As an example, the output unit 16 the tilt-corrected image TCI generated by the tilt correction unit 13; The interpolated image generated by the interpolation unit 14 Information indicating the result of the image recognition process by the image recognition unit 15 At least one of the above is visually presented to the user via the input / output unit 40, or provided to another device via the communication unit 30. (Example 1 of processing flow by information processing device 1A) An example of the flow of processing by the information processing device 1A will be described below with reference to Figures 4 to 7. Note that the arrows in Figure 4 only show an example of the direction in which data moves, and data may move in the opposite direction, or data may move between components other than those connected by arrows.

[0045] (Step S12A) First, in step S12A, the second acquisition unit 12 executes a camera attitude calculation process by referring to the camera image (image data IMG) acquired from the robot 50 by the first acquisition unit 11. In the camera attitude calculation process, as an example, the second acquisition unit 12 derives the tilt angle of the camera 55 (camera roll angle in FIG. 4). More specifically, in this step, the second acquisition unit 12 Apply image processing (edge ​​detection processing as an example) to the camera image (image data IMG), - Determine the vertical direction by referring to the detected edges, Derive the tilt angle of the camera by referring to the identified vertical tilt in the camera image (image data IMG). For example, the second acquisition unit 12 may be configured as follows: By applying edge detection processing to the camera image, the edges of product shelves and cardboard boxes contained in the camera image are detected, Derive the tilt angle of the camera by identifying the direction in which the detected edge faces in the camera image. The above configuration may also be used.

[0046] Alternatively, the second acquisition unit 12 A camera image (image data IMG) may be input to a model trained to input an image and output the tilt angle of the camera that captured the image, and the camera tilt angle may be determined by referring to the output of the model. Here, the model may be trained by referring to training data including a pair of an image and label data (teacher labels) indicating the tilt angle of the camera that captured the image, for example, but this is not intended to limit the present exemplary embodiment. Furthermore, the model may be a deep learning model such as a convolutional neural network (CNN), for example, but this is not intended to limit the present exemplary embodiment.

[0047] In addition, in this step, the second acquisition unit 12 may be configured to acquire the tilt angle of the camera by further referring to sensing data from the camera 55 or a sensor 56 (acceleration sensor or gyro sensor) attached to the gimbal 54. In the example shown in FIG. 4, the roll angle of the camera is calculated as the tilt angle of the camera by the camera attitude calculation process, but the term "roll angle" does not limit this exemplary embodiment and may include a tilt angle that is expressed as a "pitch angle."

[0048] (Step S13A) Next, in step S13A, the tilt correction unit 13 generates a tilt-corrected image TCI by applying image rotation processing to the camera image (image data IMG) acquired from the robot 50 by the first acquisition unit 11, with reference to the tilt angle of the camera 55 derived by the second acquisition unit 12.

[0049] An example 1 of the image rotation process by the tilt correction unit 13 in this step is shown in the upper part of Fig. 5. In the example shown in the upper part of Fig. 5, in the image rotation process, the camera image (imaging data IMG) is rotated by a rotation angle corresponding to the tilt angle of the camera 55, thereby generating a tilt-corrected image TCI.

[0050] As shown in the upper part of Fig. 5, the imaging target OBJ included in the camera image (imaging data IMG) was captured at an angle in the camera image (imaging data IMG), but the tilt of the imaging target OBJ is eliminated in the tilt-corrected image TCI. Meanwhile, as shown in the upper part of Fig. 5, as a result of the image rotation process according to this example, the tilt-corrected image TCI may include one or more out-of-field areas (areas R1, R2, ... in the example shown in the upper part of Fig. 5). Here, the out-of-field area refers to an area that was not included in the camera image (imaging data IMG), but this term does not limit this exemplary embodiment.

[0051] The lower part of Fig. 5 shows Example 2 of the image rotation process by the tilt correction unit 13 in this step. In the example shown in the lower part of Fig. 5, in the image rotation process, The camera image (image data IMG) is rotated by a rotation angle corresponding to the tilt angle of the camera 55, and further The tilt-corrected image TCI is generated by applying an enlargement, reduction, or cropping process to the rotated image. Here, the enlargement, reduction, or cropping process is performed, for example, so that the tilt-corrected image TCI does not include an area outside the field of view.

[0052] In the example shown in the lower part of Fig. 5, the imaging target OBJ included in the camera image (imaging data IMG) was captured tilted in the camera image (imaging data IMG), but the tilt of the imaging target OBJ is eliminated in the tilt-corrected image TCI. Also, as shown in the lower part of Fig. 5, as a result of the image rotation process according to this example, the above-mentioned out-of-field area is not included in the tilt-corrected image TCI.

[0053] 5, as a result of the enlargement, reduction, or cropping process described above, a situation may occur in which a part of the imaging target OBJ included in the camera image (imaging data IMG) is not included in the tilt-corrected image TCI. Therefore, from the viewpoint of the image recognition process described later, it can be said that Example 1 described above is more preferable than Example 2. However, this does not limit the present exemplary embodiment.

[0054] (Step S14A) Subsequently, in step S14A, the interpolation unit 14 generates an interpolated image by applying an interpolation process to the tilt-corrected image TCI generated by the tilt correction unit 13 in step S13A.

[0055] As an example, the interpolation unit 14 generates an image in which one or more out-of-field areas (areas R1, R2, ... shown in the upper part of Fig. 5) that have occurred in the tilt-corrected image TCI as a result of the image rotation process in step S13A have been interpolated, as the interpolated image. As an example, the interpolation unit 14 may input the tilt-corrected image TCI to a trained interpolation model, and obtain the interpolated image as the output of the interpolation model. Here, the interpolation model may be, for example, Training data that includes images with missing regions (also called missing images) and images without the missing regions (original images) as correct answer data The interpolation model may be a model trained using the following. The interpolation model may also have a configuration called a generative model or an encoder-decoder model. As an example of such an interpolation model, This model interpolates areas outside the field of view by performing image interpolation processing conditioned by information within the field of view contained in the camera image (image data IMG). It can also be expressed as:

[0056] In the example shown in Fig. 6, an interpolated image is generated by interpolating one or more out-of-field areas (areas R1, R2, ...) that occur in the tilt-corrected image TCI. Here, the diagonal lines in the areas R1, R2, ... in Fig. 6 indicate that the areas are occupied (filled in) by the interpolated image.

[0057] By performing the interpolation process as described above, the interpolation unit 14 can obtain, for example, The data distribution of the interpolated image can be made closer to the data distribution of the training data used for learning by the image recognition unit (described later), improving the accuracy of image recognition by the image recognition unit. - It is possible to present an interpolated image that does not look strange to the user. However, as will be described later, the interpolation process in step S14A is not essential to this exemplary embodiment, and a configuration in which the interpolation process is not performed is also included in this exemplary embodiment.

[0058] (Step S15A) In step S15A, the image recognition unit 15 the tilt-corrected image TCI generated in step S13A, and The interpolated image generated in step S14A Hereinafter, the tilt-corrected image TCI and the interpolated image referred to by the image recognition unit 15 in this step will also be referred to as target images.

[0059] As an example, in this step, the image recognition unit 15 applies an object detection process to the target image and outputs the result of the process. Here, the result of the process may be configured to include the position and type of one or more objects in the target image. Also, as an example, the image recognition unit 15 Training data including an image containing one or more objects and the position and type of the object in the image as a correct answer label The object detection process may be performed using an object detection model that has been trained with reference to the above.

[0060] Furthermore, when the image recognition unit 15 executes the image recognition process by referring to the tilt-corrected image TCI to which the interpolation process in step S14A has not been applied, the object detection model is Training data including an image containing one or more objects, the image including one or more out-of-field regions as mask regions, and the position and type of the object in the image as a correct answer label. In other words, the image recognition unit 15 may be configured to perform the image recognition process using an image recognition model that has been trained with reference to a group of images that includes a masked area corresponding to an area outside the field of view. In this way, if the image recognition process is performed using an image recognition model that has been trained with reference to a group of images that includes a masked area corresponding to an area outside the field of view, the image recognition process can be preferably performed even if the configuration does not include the interpolation process S14A described above.

[0061] (Step S16A) In step S16A, the output unit 16 presents the information derived by the above-described processes to the user via the input / output unit 40, or provides it to another device via the communication unit 30. For example, the output unit 16 the tilt-corrected image TCI generated by the tilt correction unit 13; The interpolated image generated by the interpolation unit 14 Information indicating the result of the image recognition process by the image recognition unit 15 At least one of the above is visually presented to the user via the input / output unit 40, or provided to another device via the communication unit 30.

[0062] 7 shows an example of the output information OUT that the output unit 16 displays in this step on the display provided in the input / output unit 40. As shown in FIG. 7, the output information OUT includes the following information: Tilt-corrected image TCI (or interpolated image) The object OBJ detected in the image by the object detection process A bounding box ("BB" in Figure 7) indicating the position of the object OBJ identified by the object detection process. Information indicating the type of the object OBJ identified by the object detection process ("Product A" in FIG. 7) Contains:

[0063] As mentioned above, in Example 1 of the above flow, Acquires camera images captured by a camera attached to a gimbal, The tilt angle of the camera is acquired by referring to the camera image (step S12A); A tilt correction process is applied to the camera image, with reference to the tilt angle of the camera, to generate a tilt-corrected image (step S13A). According to the above configuration, tilt that may be included in a camera image captured by a camera attached to a gimbal is corrected by referring to the camera image, so tilt correction can be performed appropriately on the camera image. As an example, tilt in a camera image caused by camera tilt that cannot be fully adjusted even using a gimbal can be appropriately corrected.

[0064] Also, in Example 1 of the above processing flow, the tilt-corrected image TCI generated in step S13A, and The interpolated image generated in step S14A Since the image recognition process is performed by referring to at least one of the above (step S15A), it is possible to derive a suitable image recognition result.

[0065] (Example 2 of processing flow by information processing device 1A) Another example of the flow of processing by the information processing device 1A will be described below with reference to Figures 8 and 9. Note that the arrows in Figures 8 and 9 only show an example of the direction in which data moves, and data may move in the opposite direction, or data may move between components other than those connected by arrows. (Step S12B) First, in step S12B, the second acquisition unit 12 executes a camera attitude calculation process by referring to at least one of the gimbal information GI and the arm joint information AJI acquired from the robot 50 by the first acquisition unit 11. In the camera attitude calculation process, as an example, the second acquisition unit 12 derives the tilt angle of the camera 55 (camera roll angle in FIG. 8). More specifically, in this step, the second acquisition unit 12 Maximum angle information MAI (limit value of the relative angle between the camera 55 and the gimbal 54) included in the gimbal information GI, Gimbal tilt information GTI (tilt of gimbal 54) included in gimbal information GI and The tilt angle of the camera 55 is derived by referring to Here, the tilt of the gimbal 54 may be derived by the control unit 511 or the control unit 20A with reference to the arm joint information AJI.

[0066] In addition, in this step, the second acquisition unit 12 may be configured to acquire the tilt angle of the camera by further referring to sensing data from the camera 55 or a sensor 56 (acceleration sensor or gyro sensor) attached to the gimbal 54.

[0067] In the example shown in FIG. 8, the roll angle of the camera is calculated as the tilt angle of the camera by the camera attitude calculation process, but the term "roll angle" does not limit this exemplary embodiment and may include a tilt angle that is expressed as a "pitch angle."

[0068] A specific example of the camera attitude calculation process in this step is shown in the upper part of Fig. 9. In the example shown in the upper part of Fig. 9, the camera attitude calculation process S12B includes a gimbal roll angle calculation process S121 and a camera roll angle calculation process S122.

[0069] (Gimbal roll angle calculation process S121) In the gimbal roll angle calculation process S121, the second acquisition unit 12 calculates the gimbal roll angle θ (the gimbal tilt information GTI described above) from the arm joint information AJI. Correspondence information indicating the correspondence between at least one of the inclination of each arm and the rotation angle of each joint included in the arm joint information AJI and the gimbal roll angle The gimbal roll angle θ is calculated by referring to the above.

[0070] (Camera roll angle calculation process S122) In the camera roll angle calculation process S122, the second acquisition unit 12 The gimbal roll angle θ calculated in the gimbal roll angle calculation process S121, Maximum applicable angle of gimbal θ max (the maximum angle information MAI mentioned above) The camera roll angle (the tilt angle of the camera 55 described above) is derived by referring to the above.

[0071] As an example, the second obtaining unit 12 executes the following steps S122-1 to S122-3 as the camera roll angle calculation process S122.

[0072] (Step S122-1) The second acquisition unit 12 acquires the gimbal roll angle θ and the maximum applicable angle θ of the gimbal. max Compared with the maximum applicable angle of the gimbal, the gimbal roll angle θ is max If it is determined that: θ≦θ max If this is true, the process proceeds to step S122-2; otherwise, the process proceeds to step S122-3.

[0073] (Step S122-2) In step S122-1, the gimbal roll angle θ is set to the maximum applicable angle θ of the gimbal. max If it is determined that the angle is equal to or smaller than the maximum applicable angle θ of the gimbal, in step S122-2, the second acquisition unit 12 derives 0 as the camera roll angle. The lower left side of FIG. 9 shows an example of the attitude of the camera 55 and the gimbal 54 corresponding to the case where 0 is derived as the camera roll angle in this step. As shown in the lower left side of FIG. 9, when the gimbal roll angle θ is equal to or smaller than the maximum applicable angle θ of the gimbal, max If it is equal to or less than this, the tilt of the camera 55 becomes 0 by the tilt adjustment function of the gimbal 54.

[0074] (Step S122-3) In step S122-1, the gimbal roll angle θ is set to the maximum applicable angle θ of the gimbal. max If it is determined that the angle is greater than θ, in step S122-3, the second acquisition unit 12 obtains θ-θ as the camera roll angle. max In the lower right of Figure 9, the camera roll angle in this step is calculated as θ-θ max9 is derived. As shown in the lower left of FIG. 9, when the gimbal roll angle θ is greater than the maximum applicable angle θ of the gimbal, max If the tilt angle is larger than θ, the tilt adjustment function of the gimbal 54 does not make the tilt of the camera 55 zero, and max is the tilt of the camera 55.

[0075] (Steps S13A to S16A) Steps S13A to S16A shown in FIG. 8 are the same as the steps explained with reference to FIG. 4, and therefore explanations thereof will be omitted here.

[0076] As described above, in Example 2 of the above processing flow, Acquires camera images captured by a camera attached to a gimbal, The tilt angle of the camera is acquired by referring to at least one of the gimbal information GI and the arm joint information AJI (step S12B). A tilt correction process is applied to the camera image, with reference to the tilt angle of the camera, to generate a tilt-corrected image (step S13A). According to the above configuration, tilt that may be included in a camera image captured by a camera attached to a gimbal is corrected by referring to at least one of the gimbal information GI and the arm joint information AJI, so that tilt correction can be suitably performed on the camera image. As an example, tilt in a camera image caused by camera tilt that cannot be fully adjusted even using a gimbal can be suitably corrected.

[0077] Furthermore, in the above processing flow example 2, similar to the above processing flow example 1, the tilt-corrected image TCI generated in step S13A, and The interpolated image generated in step S14A Since the image recognition process is performed by referring to at least one of the above (step S15A), it is possible to derive a suitable image recognition result.

[0078] (Additional Notes Regarding Example 1 and Example 2 of the Processing Flow by Information Processing Device 1A) The processes executed by the information processing device 1A are not limited to the above-described flow example 1 and flow example 2. As an example, the information processing device 1A may be configured to execute a process that combines the above-described flow example 1 and flow example 2. For example, the information processing device 1A may be configured to execute the following step S12C instead of step S12A in the above-described flow example 1 and step S12B in the above-described flow example 2.

[0079] (Step S12C) In step S12C, the second acquisition unit 12 A first candidate for the tilt angle of the camera 55 is derived by referring to the camera image (image data IMG) (step S12C-1). A second candidate for the tilt angle of the camera 55 is derived by referring to at least one of the gimbal information GI and the arm joint information AJI (step S12C-2); The tilt angle (camera roll angle) of the camera 55 is derived by referring to the first candidate and the second candidate (step S12C-3).

[0080] Here, in step S12C-1, for example, a process similar to that in step S12A described above may be performed to derive a first candidate for the tilt angle of camera 55. Also, in step S12C-2, for example, a process similar to that in step S12B described above may be performed to derive a second candidate for the tilt angle of camera 55.

[0081] In addition, in the above step S12C-3, as an example, The tilt angle (camera roll angle) of the camera 55 may be derived by taking a simple average of the first candidate and the second candidate. The tilt angle (camera roll angle) of the camera 55 may be derived by taking a weighted average of the first candidate and the second candidate. Here, when a weighted average is used, a reliability may be set in advance for each of the first candidate and the second candidate, and a weighted average may be performed using a weight according to the reliability.

[0082] (Application example) The information processing system 100A according to this exemplary embodiment can be applied to various fields, and specific application examples do not limit this exemplary embodiment, but include the following examples.

[0083] As an example, the information processing system 100A includes: - Controlling a robot arm equipped with a camera to move and inspect an object More specifically, the following can be suitably applied: - Performing warehouse inventory and stock management by controlling a robotic arm equipped with a camera In such a case, the image processing device 1 according to this exemplary embodiment can be suitably applied to the following cases: Acquires camera images captured by a camera attached to a robot arm via a gimbal; acquiring a tilt angle of the camera by referring to at least one of the camera image and gimbal information including the tilt of the gimbal; generating a tilt-corrected image by applying tilt correction processing to the camera image, the tilt angle of which is referenced; Execute image recognition processing by directly or indirectly referring to the tilt-corrected image. This allows for high accuracy in the environment recognition capability of the camera attached to the robot arm via a gimbal.

[0084] [Software implementation example] Some or all of the functions of the image processing device 1, information processing device 1A, and robot 50 (hereinafter also referred to as "each of the above devices") may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0085] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 10. Figure 10 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0086] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to operate as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0087] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0088] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0089] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0090] [Additional Notes] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0091] (Appendix A1) a first acquisition means for acquiring a camera image captured by a camera attached to a gimbal; a second acquisition means for acquiring the tilt angle of the camera by referring to at least one of the camera image and gimbal information including the tilt of the gimbal; a tilt correction means for applying a tilt correction process to the camera image, the tilt correction process referring to the tilt angle of the camera, to generate a tilt-corrected image; An image processing device comprising:

[0092] (Appendix A2) The second acquisition means Deriving the tilt angle of the camera by referring to the camera image. 10. The image processing device according to claim 1 .

[0093] (Appendix A3) The gimbal information includes: a limit value of the relative angle between the camera and the gimbal is included, The second acquisition means Derive the tilt angle of the camera by referring to the tilt of the gimbal and the limit value of the relative angle. 10. The image processing device according to claim 1, wherein the image processing device is a

[0094] (Appendix A4) The second acquisition means Deriving the inclination of the gimbal by referring to arm joint information of the robot arm to which the gimbal is attached. Image processing device described in Appendix A3

[0095] (Appendix A5) The second acquisition means The tilt angle of the camera is acquired by further referring to sensing data from an acceleration sensor or a gyro sensor attached to the camera or the gimbal. An image processing device according to any one of appendices A1 to A4.

[0096] (Appendix A6) The image processing device further includes an interpolation unit that applies an interpolation process to the tilt-corrected image to generate an interpolated image. An image processing device according to any one of appendices A1 to A5.

[0097] (Appendix A7) an image processing device according to Appendix A6; image recognition means for applying image recognition processing to the interpolated image; an output means for outputting the result of the image recognition processing; An information processing device comprising:

[0098] (Appendix A8) An image processing device according to any one of appendices A1 to A5; an image recognition unit that applies image recognition processing to the tilt-corrected image; an output means for outputting the result of the image recognition processing; An information processing device comprising:

[0099] (Appendix A9) The image recognition means The image recognition process is performed using an image recognition model that has been trained by referring to a group of images that includes a mask region corresponding to an area outside the field of view. 10. The information processing device according to claim 7,

[0100] (Appendix A10) acquiring a camera image captured by a gimbal-mounted camera; acquiring a tilt angle of the camera by referring to at least one of the camera image and gimbal information including the tilt of the gimbal; generating a tilt-corrected image by applying tilt correction processing to the camera image, the tilt angle of which is referenced; An image processing method comprising:

[0101] (Appendix A11) A program that causes a computer to function as an image processing device, The computer a first acquisition means for acquiring a camera image captured by a camera attached to a gimbal; a second acquisition means for acquiring the tilt angle of the camera by referring to at least one of the camera image and gimbal information including the tilt of the gimbal; a tilt correction means for applying a tilt correction process to the camera image, the tilt correction process referring to the tilt angle of the camera, to generate a tilt-corrected image; A program that functions as a [Explanation of symbols]

[0102] 1. Image processing device 1A Information processing equipment 11 First acquisition section 12 Second acquisition section 13 Tilt correction section 14 Interpolation section 15 Image Recognition Unit 16 Output section 100A ···Information Processing System

Claims

1. a first acquisition means for acquiring a camera image captured by a camera attached to a gimbal; a second acquisition means for acquiring the tilt angle of the camera by referring to at least one of the camera image and gimbal information including the tilt of the gimbal; a tilt correction means for applying a tilt correction process to the camera image, the tilt correction process referring to the tilt angle of the camera, to generate a tilt-corrected image; An image processing device comprising:

2. The second acquisition means Deriving the tilt angle of the camera by referring to the camera image. The image processing device according to claim 1 .

3. The gimbal information includes: a limit value of the relative angle between the camera and the gimbal is included, The second acquisition means Derive the tilt angle of the camera by referring to the tilt of the gimbal and the limit value of the relative angle. The image processing device according to claim 1 .

4. The second acquisition means Deriving the inclination of the gimbal by referring to arm joint information of the robot arm to which the gimbal is attached. The image processing device according to claim 3 .

5. The second acquisition means The tilt angle of the camera is acquired by further referring to sensing data from an acceleration sensor or a gyro sensor attached to the camera or the gimbal. The image processing device according to claim 1 .

6. The image processing device further includes an interpolation unit that applies an interpolation process to the tilt-corrected image to generate an interpolated image. The image processing device according to claim 1 .

7. The image processing device according to claim 6 ; image recognition means for applying image recognition processing to the interpolated image; an output means for outputting the result of the image recognition processing; An information processing device comprising:

8. The image processing device according to any one of claims 1 to 4, an image recognition unit that applies image recognition processing to the tilt-corrected image; an output means for outputting the result of the image recognition processing; An information processing device comprising:

9. acquiring a camera image captured by a gimbal-mounted camera; acquiring a tilt angle of the camera by referring to at least one of the camera image and gimbal information including the tilt of the gimbal; generating a tilt-corrected image by applying tilt correction processing to the camera image, the tilt angle of which is referenced; An image processing method comprising:

10. A program that causes a computer to function as an image processing device, The computer a first acquisition means for acquiring a camera image captured by a camera attached to a gimbal; a second acquisition means for acquiring the tilt angle of the camera by referring to at least one of the camera image and gimbal information including the tilt of the gimbal; a tilt correction means for applying a tilt correction process to the camera image, the tilt correction process referring to the tilt angle of the camera, to generate a tilt-corrected image; A program that functions as a

Images