Image processing device and image processing method

Abstract

The objective of the present invention is to make it possible to generate a display image having little sense of incongruity, without increasing an amount of processing.　A 360-degree image generating unit generates a rectangular cuboid shaped 360-degree image on the basis of a plurality of captured images. For example, boundary portions are provided by means of blurring, for example, in joining parts between surfaces of the rectangular cuboid shaped 360-degree image. A display image generating unit generates a display image in a prescribed direction on the basis of the 360-degree image. The display image is displayed by a display unit, or is delivered to an end user by means of a delivery unit.

Description

[Technical Field]

[0001] The present technology relates to an image processing device and an image processing method, and more particularly to an image processing device and an image processing method that generate a 360-degree image and obtain an image for display in a predetermined direction based on the image. [Background technology]

[0002] For example, Patent Document 1 discloses a technology for synthesizing and displaying images captured by cameras on the front, rear, left, and right sides of a vehicle. The display image (synthesized image) in this case is intended to be displayed on an in-vehicle display, and high resolution is not required. Synthesizing captured images at high resolution poses a problem of high processing load. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-127171 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present technology is to enable the generation of a display image that does not appear unnatural without increasing the amount of processing. [Means for solving the problem]

[0005] The concept of this technology is: a 360-degree image generation unit that generates a rectangular parallelepiped 360-degree image based on a plurality of captured images; a display image generating unit that generates a display image in a predetermined direction based on the 360-degree image; It is in the image processing device.

[0006] In this technology, a 360-degree image generating unit generates a rectangular parallelepiped 360-degree image based on a plurality of captured images. For example, the plurality of captured images may be four captured images obtained by four imaging units attached to the exterior of a vehicle, each capturing an image of the front, right, left, and rear, respectively. In this case, it is possible to generate a 360-degree image with the vehicle at its center as the rectangular parallelepiped 360-degree image.

[0007] In this case, for example, the 360-degree image generation unit may be configured to attach an image obtained by an imaging unit that captures a front view to a front region of the rectangular parallelepiped, an image obtained by an imaging unit that captures a rear view to a rear region of the rectangular parallelepiped, an image obtained by an imaging unit that captures a right view to a right region of the rectangular parallelepiped and to the right half regions of each of the upper and lower surfaces adjacent to the right region, and an image obtained by an imaging unit that captures a left view to a left region of the rectangular parallelepiped and to the left half regions of each of the upper and lower surfaces adjacent to the left region, thereby generating a rectangular parallelepiped 360-degree image. This makes it possible to easily create a 360-degree image centered on the vehicle based on four captured images from the front, rear, left, and right.

[0008] In this case, for example, the imaging units capturing images of the front and rear may be imaging units capturing wide-angle images, and the imaging units capturing images of the right and left may be imaging units capturing fisheye images. This allows the captured images obtained by the four imaging units (front, rear, left, and right) to capture a wider range, making it possible to include an image of a wider range as a 360-degree image centered on the vehicle. Here, the 360-degree image generation unit may be configured to crop and use the center portion of each of the four captured images. This makes it possible to generate a 360-degree image with less distortion.

[0009] In this case, for example, a receiving unit that receives four captured images from the vehicle may be further provided, which makes it possible to receive and process the four captured images from the vehicle at a location away from the vehicle.

[0010] The display image generation unit generates a display image in a predetermined direction based on the 360-degree image. For example, the display image generation unit may generate the display image in the predetermined direction by capturing a rectangular 360-degree image with a virtual camera with a predetermined angle of view, installed at the center of the rectangular 360-degree image, and set in a predetermined direction. This makes it possible to easily generate a display image in the predetermined direction based on the 360-degree image.

[0011] Furthermore, for example, a selection unit that selects a predetermined direction based on a user operation may be further provided. In this case, since the predetermined direction can be selected based on the user operation, it is possible to obtain a display image in any direction.

[0012] Also, for example, a distribution unit that distributes display images may be further provided. This makes it possible to distribute display images. Also, a display unit that displays display images may be further provided. This makes it possible to display and check the display images.

[0013] In this way, this technology generates a rectangular parallelepiped 360-degree image based on multiple captured images, and then generates a display image in a predetermined direction based on this 360-degree image.Compared to generating a display image in a predetermined direction based on a spherical or cylindrical 360-degree image obtained by performing distortion correction processing, stitching processing, etc., this technology makes it possible to obtain a display image that is less unnatural without increasing the amount of processing.In addition, because the processing load is small, processing of each frame of the video can be easily performed in real time.

[0014] In the present technology, for example, the 360-degree image generation unit may provide a boundary at the joint between the faces of a rectangular parallelepiped 360-degree image. This makes it possible to reduce the sense of incongruity at the joint between the captured images, even when a rectangular parallelepiped 360-degree image is generated by simply pasting captured images from different image capture units onto each face. In this case, for example, the boundary may be generated by blurring the images of the two faces corresponding to the joint. This makes it possible to more naturally reduce the sense of incongruity at the joint between the captured images. In addition, in the present technology, for example, the multiple captured images may be captured images after an identification process has been applied to the area of ​​a specific object so that the object cannot be identified. This makes it possible to prevent the identification of a specific object, such as a person, and thereby protect privacy.

[0015] Another concept of the present technology is generating a rectangular parallelepiped 360-degree image based on a plurality of captured images; generating an image for display in a predetermined direction based on the 360-degree image; The image processing method. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram showing an example of the appearance of a vehicle according to the present technology; [Figure 2] FIG. 10 is a diagram showing how images captured by four cameras on the front, rear, left and right sides are transmitted from a transmitter on a vehicle to a remote monitoring device. [Figure 3] FIG. 2 is a block diagram showing an example of the configuration of a remote monitoring device. [Figure 4] FIG. 1 is a diagram showing an example of an image captured by four cameras on the front, rear, left and right sides, and an example of each cutout range. [Figure 5] FIG. 10 is a diagram for explaining generation of a 360-degree image by pasting images. [Figure 6] FIG. 10 is a diagram illustrating an example of a boundary portion. [Figure 7]FIG. 10 is a diagram schematically showing an example of a display image (two-dimensional image) when the predetermined direction is selected to be the front upper direction. [Figure 8] FIG. 10 is a diagram showing an example of a display image of the upper front part of a 360-degree image. [Figure 9] FIG. 10 is a diagram showing an example of a display image for the lower rear of a 360-degree image. [Figure 10] 10 is a flowchart illustrating an example of a processing procedure for generating a display image in a remote monitoring device and displaying and distributing the image. [Figure 11] 10A and 10B are diagrams illustrating an example of performing identification processing on a region of a specific object so that the specific object is not identified when the specific object is included in a captured image after clipping. [Figure 12] 10 is a flowchart illustrating another example of a processing procedure for generating a display image in a remote monitoring device and displaying and distributing the image. [Figure 13] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0017] The following describes modes for carrying out the invention (hereinafter referred to as "embodiments") in the following order: 1. Embodiment 2. Variations

[0018] <1. Embodiment> FIG. 1 shows an example of the appearance of a vehicle 10 according to the present technology. The vehicle 10 is an autonomous vehicle or a remotely controlled vehicle, and is not provided with windows. A display 11 is provided on the exterior of the vehicle 10. This display 11 is provided for displaying advertisements and the like.

[0019] Furthermore, cameras 12a, 12b, 12c, and 12d are attached to the front, right side, left side, and rear of the exterior of vehicle 10 as imaging units for capturing images of the front, right, left, and rear, respectively.

[0020] In this embodiment, camera 12a capturing images of the front and camera 12d capturing images of the rear are each a wide-angle image capturing unit equipped with a wide-angle lens, and camera 12b capturing images of the right and camera 12c capturing images of the left are each a fisheye image capturing unit equipped with a fisheye lens. As a result, the images captured by the four cameras 12a, 12b, 12c, and 12d on the front, rear, left, and right sides capture a wider range, and it is possible to include an image of a wider range as a 360-degree image centered on vehicle 10, which will be described later.

[0021] In addition, although not shown, a transmitting device is arranged inside the vehicle 10 for transmitting the four images captured by the cameras 12a, 12b, 12c, and 12d to a remote monitoring device that constitutes the image processing device of the present technology.

[0022] 2 shows that four captured images obtained by cameras 12a, 12b, 12c, and 12d are transmitted from transmitter 13 of vehicle 10 to remote monitoring device 100. Transmitting device 13 transmits, for example, four captured images input from cameras 12a, 12b, 12c, and 12d, namely, a front image Ia, a right image Ib, a left image Ic, and a rear image Id, to remote monitoring device 100, for example, via a telephone line. In this case, the resolution of the four captured images is limited depending on the communication capacity of the line.

[0023] 3 shows an example of the configuration of the remote monitoring device 100. This remote monitoring device 100 has a control unit 101, a user operation unit 102, a receiving unit 103, an image cropping unit 104, an image pasting unit 105, a boundary portion processing unit 106, a display image generating unit 107, a display unit 108, and a distribution unit 109.

[0024] The control unit 101 controls the operation of each unit of the remote monitoring device 100. The user operation unit 102 is a part where the user issues various operation instructions. These operations include the operation of selecting a predetermined direction that is required when generating a display image from a 360-degree image.

[0025] The receiving unit 103 receives the captured images Ia, Ib, Ic, and Id captured by the cameras 12a, 12b, 12c, and 12d sent from the transmitting device 13 (see FIG. 2) of the vehicle 10. The image cropping unit 104 crops out the central portion, i.e., the portion with least distortion, of each of the captured images Ia, Ib, Ic, and Id received by the receiving unit 103, and outputs the cropped captured image Ia. ´ ,Ib ´ ,I C ´ ,Id ´ By performing the cutout in this manner, it is possible to generate a 360-degree image with less distortion, centered on the vehicle 10, which will be described later.

[0026] 4 shows examples of captured images Ia, Ib, Ic, and Id captured by cameras 12a, 12b, 12c, and 12d, and examples of their respective cropping ranges. In the illustrated example, the cropping ranges are indicated by rectangular frames. Note that the cropping ranges are fixed, for example, but may be changeable in size and position in response to user operations.

[0027] Returning to FIG. 3, the image pasting unit 105 applies the captured image Ia obtained by the image cutting unit 104 to the ´ ,Ib ´ ,I C ´ ,Id ´ The images are attached to each surface of the rectangular parallelepiped to generate a rectangular parallelepiped 360-degree image. Figure 5(a) shows the captured image Ia after cutting out the front, right, left, and rear. ´ ,Ib ´ ,I C ´ ,Id ´ FIG. 5(b) shows the rectangular parallelepiped unfolded, and the captured image Ia ´ ,Ib ´ ,I C ´ ,Id ´ The figure shows the area where each of the above will be pasted.

[0028] In this case, the front captured image Ia ´ is attached to the front of the rectangular parallelepiped. ´The right-side image Ib' is attached to the right side of the rectangular parallelepiped and to the right half of the top and bottom surfaces adjacent to the right side. c´ is a rectangular parallelepiped left Surface position and this left Above adjacent face position Face The left half of the top and bottom surfaces of the camera are attached to the camera. ´ ,Ib ´ ,I C ´ ,Id ´ Based on this, a 360-degree image centered on the vehicle 10 can be easily created.

[0029] This 360-degree image is, as shown in FIG. 5(c), an image Ia of the front of the vehicle 10 projected in front of the rectangular parallelepiped when the user places his / her head inside the rectangular parallelepiped. ´ The right half of the upper surface, the right side, and the right half of the lower surface of the rectangular parallelepiped are shown with the captured image Ib ´ The left half of the upper surface, the left surface, and the left half of the lower surface of the rectangular parallelepiped are shown with the captured image Ic ´ Furthermore, the rear surface of the rectangular parallelepiped is provided with a captured image Id of the rear of the vehicle 10. ´ It becomes something that can be observed.

[0030] Returning to Fig. 3, boundary portion processing unit 106 provides a boundary portion at the joint between the faces of the rectangular parallelepiped 360-degree image generated by image pasting unit 105. This boundary portion satisfies the following requirements, for example: the scenery can be clearly seen, the scenery is not obstructed, and the unnaturalness of the joints can be eliminated.

[0031] FIG. 6 shows examples of boundary portions. FIG. 6(a) is an example in which a window frame 110a is provided as the boundary portion. FIG. 6(b) is an example in which a frame 110b is provided as the boundary portion. FIG. 6(c) is an example in which a line (solid line) 110c is provided as the boundary portion. FIG. 6(d) is an example in which a dotted line 110d is provided as the boundary portion. FIG. 6(e) is an example in which a color pick line 110e is provided as the boundary portion. FIG. 6(f) is an example in which a blur processing unit 110f is provided that performs blur processing using images of two surfaces corresponding to the joint portion as the boundary portion.

[0032] By providing a boundary in this way, it is possible to reduce the sense of incongruity at the joints between the captured images. In particular, by using blur processing to create a boundary, it is possible to more naturally reduce the sense of incongruity at the joints between the captured images.

[0033] In the rectangular parallelepiped 360-degree image, the right half of the upper surface, the right surface, and the right half of the lower surface show the captured image Ib of the right side of the vehicle 10. ´ Similarly, in the rectangular parallelepiped 360-degree image, the left half of the upper surface, the left half of the lower surface, and the left half of the upper surface are pasted with the image Ic of the left side of the vehicle 10. ´ are pasted, and the boundary between the top and left surfaces and the boundary between the left and bottom surfaces are not seams between the captured images.

[0034] In this way, by providing a boundary even if it is not at the seam between captured images, the sense of incongruity of a 360-degree image can be reduced. In this case, the captured images on the right and left are fisheye images that also include images above and below, and these images above and below are pasted onto the top and bottom surfaces of the rectangular parallelepiped. The human eye has a narrow vertical field of view, so when looking at the images on the right or left, the images above and below cannot be seen. In other words, it is unnatural for the images above and below, which are not visible when looking at the images on the right or left normally, to be connected, and providing a boundary can reduce this sense of incongruity.

[0035] As described above, boundary portion processing unit 106 not only provides a boundary portion at the joint between the faces of the rectangular parallelepiped 360-degree image generated by image pasting unit 105, but also provides a similar boundary portion at the joint between the right captured image and the left captured image of the upper and lower faces, thereby reducing the sense of incongruity at these joints.

[0036] Returning to FIG. 3, the display image generation unit 107 generates a display image (two-dimensional image) in a predetermined direction based on the 360-degree image after the boundary portion processing unit 106 has provided the boundary portion. This predetermined direction can be selected, for example, by a selection operation using the user operation unit 102. The display image generation unit 107 generates a display image in the predetermined direction, for example, by performing a process of capturing a rectangular parallelepiped 360-degree image with a virtual camera with a predetermined angle of view that is installed at the center position of the rectangular parallelepiped (see the head position in FIG. 5(c)) and has an imaging direction set in the predetermined direction. FIG. 7 schematically shows an example of a display image when, for example, the predetermined direction is selected to be upward in front.

[0037] 8 and 9 show examples of display images in the case where the captured images Ia, Ib, Ic, and Id of cameras 12a, 12b, 12c, and 12d (see FIG. 1) attached to the front, right side, left side, and rear exterior of vehicle 10 are shown in FIG. 4. FIG. 8 shows a display image when the predetermined direction is selected as the upper front, and FIG. 9 shows a display image when the predetermined direction is selected as the lower rear. Note that the boundaries in these display images are formed by blurring.

[0038] Returning to Fig. 3, the display unit 108 is configured, for example, with a liquid crystal display panel or an organic EL panel, and displays the display image generated by the display image generation unit 107. This allows the user of the remote monitoring device 100 to check the display image, i.e., the image of the surroundings of the vehicle 10. In this case, when changing the predetermined direction in which the display image is generated, the direction of the display image generated by the display image generation unit 107 can be changed, and therefore the user can check the image of the surroundings of the vehicle 10 in any direction.

[0039] The distribution unit 109 distributes the display image generated by the display image generation unit 107. This allows the end user (the user to whom the image is distributed) to check the display image, that is, the image of the surroundings of the vehicle 10. In this case, by making it possible to change the predetermined direction in which the display image is generated according to an instruction from the end user, the end user can check the image of the surroundings of the vehicle 10 in any direction.

[0040] The flowchart in FIG. 10 shows an example of a processing procedure for generating a display image in the remote monitoring device 100 shown in FIG. 3 and then displaying and distributing the image.

[0041] First, in step ST1, the remote monitoring device 100 receives, at the receiving unit 103, captured images Ia, Ib, Ic, and Id of the front, right, left, and rear of the vehicle 10 captured by the cameras 12a, 12b, 12c, and 12d, which are sent from the transmitting device 13 of the vehicle 10 (see Figure 2).

[0042] Next, in step ST2, the remote monitoring device 100 causes the image cutout unit 104 to cut out the central portion, that is, the portion with least distortion, from each of the captured images Ia, Ib, Ic, and Id.

[0043] Next, in step ST3, the remote monitoring device 100 uses the image pasting unit 105 to past the cut-out captured images Ia of the front, right, left, and rear. ´ ,Ib ´ ,I C ´ ,Id ´ is attached to each face of a rectangular parallelepiped to generate a 360-degree rectangular parallelepiped image.

[0044] Next, in step ST4, the remote monitoring device 100 uses the boundary part processing unit 106 to create a boundary part, such as by performing blur processing using two corresponding images at the joints between the faces of the rectangular 360-degree image.

[0045] Next, in step ST5, the remote monitoring device 100 causes the display image generating unit 107 to generate a display image (two-dimensional image) in a predetermined direction based on the rectangular parallelepiped 360-degree image with a boundary.

[0046] Next, in step ST6, the remote monitoring device 100 displays the generated display image on the display unit 108 and distributes it through the distribution unit 109.

[0047] In the above example, specific objects included in the captured images Ia, Ib, Ic, and Id, such as people, address signs, and advertising billboards, are distributed as they are. However, from the viewpoint of privacy protection, it is also possible to distribute the specific objects after performing an identification process that makes them unidentifiable.

[0048] In this case, it is possible to identify the area of ​​a specific object using a segmentation process that uses pattern matching or a learning model, and then perform a specific process on this area to prevent the specific object from being identified, such as blurring, blacking out, or replacing it with a character.

[0049] 11 shows examples of captured images Ia, Ib, Ic, and Id. For example, captured image Ia includes a person M as a specific object. In this case, in the illustrated example, the area of ​​person M in captured image Ia is identified, as shown by an enlarged view of the area of ​​person M, and blurring processing is performed as identification processing to prevent person M from being identified. Although detailed explanation will be omitted, similar processing is also performed on specific objects included in the other captured images Ib, Ic, and Id.

[0050] As described above, the captured images Ia, Ib, Ic, and Id are not distributed as they are, but rather, portions with less distortion are cut out from the captured images Ia, Ib, Ic, and Id, and the cut-out captured images Ia ´ ,Ib ´ ,I C ´ ,Id ´Therefore, in reality, the captured image Ia after the cutout is delivered (see step ST2 in FIG. 4 and FIG. 10). ´ ,Ib ´ ,I C ´ ,Id ´ It is then determined whether a specific object is included in the image, and if so, a specific process is performed on the area of ​​the specific object so that the specific object is not identified.

[0051] The flowchart of Fig. 12 shows another example of the processing procedure from generating a display image to displaying and distributing it in the remote monitoring device 100 shown in Fig. 3. In this flowchart, processing of step ST2a is added between steps ST2 and ST3 in the flowchart of Fig. 10, and although detailed description will be omitted, the rest is the same as the flowchart of Fig. 10.

[0052] In step ST2a, the remote monitoring device 100 outputs the captured image Ia cut out by the image cutout unit 104. ´ ,Ib ´ ,I C ´ ,Id ´ If a specific object such as a person is included in the image, the area of ​​the specific object is identified, and an identification process is performed on the area so that the specific object is not identified.

[0053] In this case, the captured images Ia of the front, right, left, and rear directions cut out in step ST3 ´ ,Ib ´ ,I C ´ ,Id ´is a captured image after identification processing has been performed so that the specific object is not identified. Although detailed description will be omitted, in correspondence with the flowchart of Fig. 12, in the remote monitoring device 100 shown in Fig. 3, between the image cropping unit 104 and the image pasting unit 105, there is a processing unit that identifies the area of ​​the specific object and performs identification processing on that area so that the specific object is not identified. Note that the function of this processing unit may be configured to be included in the image cropping unit 104 or the image pasting unit 105.

[0054] Furthermore, the user may be able to arbitrarily select what to select as the specific object, for example, whether to select only a person. In the case of a person, it is also conceivable to apply a specific processing to only the face area so that it is not identified. Furthermore, the user may be able to select whether or not to apply a specific processing to prevent the specific object from being identified.

[0055] "Software processing" The process shown in the flowchart of Figure 10 can be executed by hardware or software. When a series of processes is executed by software, the program constituting the software is installed from a recording medium into a computer incorporated in dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0056] figure 13 is a block diagram showing an example of the hardware configuration of a computer 400. The computer 400 has a CPU 401, a ROM 402, a RAM 403, a bus 404, an input / output interface 405, an input unit 406, an output unit 407, a storage unit 408, a drive 409, a connection port 410, and a communication unit 411. Note that the hardware configuration shown here is an example, and some of the components may be omitted. Furthermore, the computer 400 may further include components other than those shown here.

[0057] The CPU 401 functions as, for example, an arithmetic processing device or a control device, and controls all or part of the operations of the components based on various programs recorded in the ROM 402 , RAM 403 , storage unit 408 or removable recording medium 501 .

[0058] The ROM 402 is a means for storing programs to be read into the CPU 401, data to be used for calculations, etc. The RAM 403 temporarily or permanently stores, for example, the programs to be read into the CPU 401 and various parameters that change as appropriate when the programs are executed.

[0059] The CPU 401, ROM 402, and RAM 403 are connected to one another via a bus 404. On the other hand, various components are connected to the bus 404 via an interface 405.

[0060] The input unit 406 may include, for example, a mouse, a keyboard, a touch panel, a button, a switch, a lever, etc. Furthermore, the input unit 406 may also include a remote controller (hereinafter referred to as a remote control) that can transmit control signals using infrared rays or other radio waves.

[0061] The output unit 407 is a device capable of visually or audibly notifying the user of the acquired information, such as a display device such as a CRT (Cathode Ray Tube), LCD, or organic EL, an audio output device such as a speaker or headphones, a printer, a mobile phone, or a facsimile.

[0062] The storage unit 408 is a device for storing various types of data. For example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device may be used as the storage unit 408.

[0063] The drive 409 is a device that reads information recorded on a removable recording medium 501 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable recording medium 501 .

[0064] The removable recording medium 501 is, for example, a DVD medium, a Blu-ray (registered trademark) medium, an HD DVD medium, various semiconductor storage media, etc. Of course, the removable recording medium 501 may also be, for example, an IC card equipped with a contactless IC chip, an electronic device, etc.

[0065] The connection port 410 is a port for connecting an external device 502, such as a Universal Serial Bus (USB) port, an IEEE 1394 port, a Small Computer System Interface (SCSI), an RS-232C port, or an optical audio terminal. The external device 502 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

[0066] The communication unit 411 is a communication device for connecting to the network 503, such as a communication card for wired or wireless LAN, Bluetooth (registered trademark), or WUSB (Wireless USB), a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

[0067] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0068] As explained above, the remote monitoring device 100 shown in Fig. 3 generates a rectangular parallelepiped 360-degree image based on four captured images of the front, rear, left, and right sides of the vehicle 10, and then generates a display image for a predetermined direction based on this 360-degree image, which makes it possible to obtain a display image (2D image) that looks less strange without increasing the amount of processing, compared to generating a display image for a predetermined direction based on a spherical or cylindrical 360-degree image obtained by performing distortion correction processing, stitching processing, etc. Furthermore, because the processing load is small, processing of each frame of a video can be easily performed in real time.

[0069] Furthermore, in the remote monitoring device 100 shown in FIG. 3, when generating a rectangular parallelepiped 360-degree image, boundaries are provided at the joints between faces, etc., so that even when a rectangular parallelepiped 360-degree image is generated by simply attaching captured images from different directions to each face, it is possible to reduce the sense of incongruity at the joints between captured images.

[0070] <2. Modifications> In the above-described embodiment, an example has been shown in which a rectangular parallelepiped 360-degree image is generated based on captured images obtained by four cameras 12a to 12d attached to the front, rear, left, and right sides of the vehicle 10. However, in the present technology, the number of captured images is not limited to four, and a rectangular parallelepiped 360-degree image may be generated based on other numbers of captured images. For example, it is also conceivable to generate a rectangular parallelepiped 360-degree image based on captured images (e.g., fisheye images) captured by two cameras, one facing the front right and one facing the rear left.

[0071] Furthermore, in the above-described embodiment, an example of processing captured images obtained by a camera attached to the vehicle 10 is shown, but similar examples of processing captured images obtained by a camera attached to other mobile devices such as drones and robots are also conceivable. In such cases, the user can also remotely monitor images of the surroundings of the mobile device.

[0072] In the above-described embodiment, an example has been shown in which a display image (two-dimensional image) of a predetermined direction generated based on a rectangular parallelepiped 360-degree image is distributed from the distribution unit 109 to an end user. However, a configuration in which a rectangular parallelepiped 360-degree image is distributed to an end user is also conceivable. In this case, the end user uses a predetermined image processing application to generate and display a display image of a predetermined direction from the rectangular parallelepiped 360-degree image. Furthermore, a configuration in which a display image (two-dimensional image) of a predetermined direction generated by the remote monitoring device 100 is distributed to the vehicle 10 and displayed on a display unit provided in the cabin of the vehicle 10 is also conceivable. In this case, the occupants of the vehicle 10 can view the display image of the 360-degree direction while facing the direction of travel.

[0073] Furthermore, while the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0074] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0075] The present technology can also be configured as follows. (1) a 360-degree image generation unit that generates a rectangular parallelepiped 360-degree image based on a plurality of captured images; a display image generating unit that generates a display image in a predetermined direction based on the 360-degree image; Image processing device. (2) The 360-degree image generating unit provides a boundary portion at a joint between the surfaces of the rectangular parallelepiped 360-degree image. The image processing device according to (1) above. (3) The boundary is generated by blurring the images of the two surfaces corresponding to the joint. The image processing device according to (2) above. (4) The display image generation unit generates the display image in the predetermined direction by performing processing to capture the rectangular parallelepiped 360-degree image with a virtual camera installed at the center position of the rectangular parallelepiped and having a predetermined angle of view, with the imaging direction set in the predetermined direction. The image processing device according to any one of (1) to (3). (5) A selection unit that selects the predetermined direction based on a user operation is further provided. The image processing device according to any one of (1) to (4). (6) A distribution unit that distributes the display image is further provided. The image processing device according to any one of (1) to (5). (7) A display unit for displaying the display image is further provided. The image processing device according to any one of (1) to (6). (8) The plurality of captured images are four captured images obtained by four imaging units attached to the exterior of the vehicle, each capturing an image of the front, right, left, and rear. The image processing device according to any one of (1) to (7). (9) The 360-degree image generating unit pastes the captured image obtained by the imaging unit that captures the front image onto a front region of the rectangular parallelepiped, pastes the captured image obtained by the imaging unit that captures the rear image onto a rear region of the rectangular parallelepiped, pastes the captured image obtained by the imaging unit that captures the right image onto a right surface region of the rectangular parallelepiped and onto right half regions of the upper and lower surfaces adjacent to the right surface region, and pastes the captured image obtained by the imaging unit that captures the left image onto a left surface region of the rectangular parallelepiped and onto left half regions of the upper and lower surfaces adjacent to the left surface region, thereby generating the rectangular parallelepiped 360-degree image. The image processing device according to (8) above. (10) The imaging units for capturing images of the front and rear are imaging units for capturing wide-angle images, The imaging units for capturing images of the right and left sides are fisheye image imaging units. The image processing device according to (8) or (9). (11) The 360-degree image generator cuts out and uses the center portion of each of the four captured images. The image processing device according to (10) above. (12) The system further includes a receiving unit that receives the four captured images from the vehicle. The image processing device according to any one of (8) to (11). (13) The plurality of captured images are captured images after a specific object area has been subjected to a specific processing so that the specific object is not identified. The image processing device according to any one of (1) to (12). (14) generating a rectangular parallelepiped 360-degree image based on a plurality of captured images; generating an image for display in a predetermined direction based on the 360-degree image; Image processing methods. [Explanation of symbols]

[0076] 10. Vehicle 12a~12d···Camera 13. Transmitting device 100 Remote monitoring device 101 Control unit 102 User operation unit 103 Receiving unit 104 Image extraction section 105...Image pasting section 106 Boundary part processing unit 107...Display image generation unit 108...Display section 109···Distribution Department

Claims

1. a 360-degree image generating unit that generates a rectangular parallelepiped 360-degree image by attaching a plurality of captured images to each surface of a rectangular parallelepiped; a display image generating unit that generates a display image in a predetermined direction based on the 360-degree image; Image processing device.

2. The 360-degree image generating unit provides a boundary portion at a joint between the surfaces of the rectangular parallelepiped 360-degree image. The image processing device according to claim 1 .

3. The boundary portion is generated by blurring using the images of the two surfaces corresponding to the joint portion. The image processing device according to claim 2 .

4. The display image generation unit generates the display image in the predetermined direction by performing processing to capture the rectangular parallelepiped 360-degree image with a virtual camera that is installed at the center position of the rectangular parallelepiped and has an imaging direction set in the predetermined direction and a predetermined angle of view. The image processing device according to claim 1 .

5. The apparatus further includes a selection unit that selects the predetermined direction based on a user operation. The image processing device according to claim 1 .

6. a distribution unit that distributes the display image The image processing device according to claim 1 .

7. a display unit for displaying the display image The image processing device according to claim 1 .

8. The plurality of captured images are four captured images obtained by four imaging units attached to the exterior of the vehicle, each capturing an image of the front, right, left, and rear directions, respectively. The image processing device according to claim 1 .

9. The 360-degree image generating unit pastes the captured image obtained by the imaging unit that images the front onto a front region of a rectangular parallelepiped, pastes the captured image obtained by the imaging unit that images the rear onto a rear region of the rectangular parallelepiped, pastes the captured image obtained by the imaging unit that images the right onto a right surface region of the rectangular parallelepiped and onto right half regions of the upper surface and lower surface adjacent to the right surface region, and pastes the captured image obtained by the imaging unit that images the left onto a left surface region of the rectangular parallelepiped and onto left half regions of the upper surface and lower surface adjacent to the left surface region, thereby generating the rectangular parallelepiped 360-degree image. The image processing device according to claim 8 .

10. the imaging units for capturing images of the front and rear are imaging units for capturing wide-angle images, The imaging units for capturing images of the right and left sides are fisheye image imaging units. The image processing device according to claim 8 .

11. The 360-degree image generating unit cuts out and uses the center portion of each of the four captured images. The image processing device according to claim 10.

12. a receiving unit that receives the four captured images from the vehicle The image processing device according to claim 8 .

13. The plurality of captured images are captured images after a specific object area has been subjected to a specific processing so that the specific object is not identified. The image processing device according to claim 1 .

14. A procedure for generating a 360-degree rectangular parallelepiped image by attaching a plurality of captured images to each surface of a rectangular parallelepiped; generating a display image in a predetermined direction based on the 360-degree image; Image processing methods.

Images