Image processing apparatus, display device, control method, storage medium, and computer program product

Abstract

The invention relates to an image processing apparatus, a display device, a control method, a storage medium, and a computer program product. The image processing apparatus may complete image processing at a frame rate at the time of streaming. The image processing apparatus is communicatively connected to a display device that displays a left-eye image viewable by the left eye and a right-eye image viewable by the right eye. The image processing apparatus includes: a memory device that stores instructions; and a processor that executes the instructions to: obtain a fisheye image, the fisheye image being a moving image captured using a fisheye lens; in a case where the fisheye image includes the left eye image and the right eye image, performing image processing on one of the left eye image and the right eye image to reduce distortion; and performing the image processing for another image in a case where the image processing for the one image is completed within a predetermined period of time.

Description

TECHNICAL FIELD

[0001] Aspects of embodiments relate to an image processing apparatus connectable to a display apparatus, the display apparatus, a control method of the image processing apparatus, and a storage medium storing a control program for the image processing apparatus. BACKGROUND

[0002] The spread of the Internet has made it possible to live stream a moving image in real time. Further, a camera in which two fisheye lenses are installed left and right is known. The fisheye lens has a field of view angle of 180 degrees. Such a camera can capture a VR180 moving image. “VR180” is a standard for a VR moving image of which a field of view angle for stereoscopic viewing is 180 degrees. When a VR180 moving image is VR live streamed in real time using such a camera, it is necessary to convert a fisheye moving image obtained by each fisheye lens into a moving image in an equirectangular projection format in real time according to a frame rate at the time of streaming.

[0003] The VR180 moving image is streamed to a head-mounted display and displayed in a manner that is stereoscopically visible. When a VR180 moving image displayed on a head-mounted display is visually recognized, a parallax between a left image and a right image is corrected. For example, since the correction uses meta information of a camera and a lens, the correction tends to be performed by a streaming source of the VR180 moving image. Japanese Patent Application Publication No. 2012-100256 (JP 2012-100256 A) discloses a system having an apparatus that divides one screen and transmits a left-eye image and a right-eye image when 3D broadcasting is performed, and an apparatus that alternately displays each image in time.

[0004] However, when a VR180 moving image is VR live streamed in real time, the system described in the above publication can not be able to complete image processing for the left-eye image and the right-eye image at a frame rate at the time of streaming. SUMMARY

[0005] The present disclosure provides an image processing apparatus, a display apparatus, a control method of the image processing apparatus, and a storage medium storing a control program, which are capable of completing image processing at a frame rate at the time of streaming.

[0006] Accordingly, one aspect of an embodiment provides an image processing apparatus that is communicably connected to a display device that displays a left-eye viewable left-eye image and a right-eye viewable right-eye image. The image processing apparatus includes a memory device that stores a set of instructions and at least one processor that executes the set of instructions to obtain a fisheye image that is a moving image taken using a fisheye lens, perform image processing for one of the left-eye image and the right-eye image to reduce distortion in a case where the fisheye image includes the left-eye image and the right-eye image, and perform the image processing for the other image in a case where the image processing for the one image is completed within a predetermined period of time.

[0007] Features of the present disclosure will become apparent from the following description of embodiments with reference to the drawings. The following description of embodiments is described by way of example with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a schematic configuration diagram illustrating an example of the overall structure of an image processing system.

[0009] Figure 2 is a block diagram illustrating an example of a hardware structure of an image processing apparatus.

[0010] Figure 3 is a flowchart illustrating a process performed by an image processing apparatus.

[0011] Figure 4A is a diagram for describing the process in steps S305 and S309 of the flowchart illustrated in Figure 3

[0012] Figure 4B is a diagram for describing the process in steps S308 and S310 of the flowchart illustrated in Figure 3

[0013] Figure 5 is a flowchart illustrating a process performed by an HMD.

[0014] Figure 6A is a diagram for describing the process in step S503 of the flowchart illustrated in Figure 5

[0015] Figure 6B is a diagram for describing the process in step S505 of the flowchart illustrated in Figure 5 DETAILED DESCRIPTION

[0016] ​​​​Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. However, the structures described in the following embodiments are merely examples, and the scope of the present disclosure is not limited to the structures described in the embodiments. For example, each unit constituting the present disclosure can be replaced with any unit capable of exhibiting the same function. Furthermore, any constituent element can be added.

[0017] Figure 1 is a schematic configuration diagram showing an example of the overall structure of an image processing system. As shown in Figure 1 , the image processing system (information processing system) 1000 includes a digital camera 100, an image processing device 110, and an HMD (Head Mounted Display) 120. A lens unit 105 is detachably mounted on the digital camera 100. The lens unit 105 is a double fisheye lens (VR180 lens) unit having two fisheye lenses. The digital camera 100 can capture a still image or a moving image as a double fisheye image (circular fisheye image) having parallax, with the lens unit 105.

[0018] In the present embodiment, the moving image is a processing target in the image processing device 110. The double fisheye image includes a left eye image (left eye circular fisheye image) viewable by a left eye of a user 130 wearing the HMD 120 and a right eye image (right eye circular fisheye image) viewable by a right eye of the user 130. The image processing device 110 is communicably coupled to the digital camera 100 via an HDMI (registered trademark) cable 115. Therefore, the image processing device 110 can obtain the double fisheye image captured by the digital camera 100 (obtaining step). Then, the image processing device 110 can perform image processing on the double fisheye image to reduce distortion, and can output the right eye image and the left eye image, which have been reduced in distortion by the image processing, to the HMD 120.

[0019] The image processing device 110 is not particularly limited, and for example, a desktop or notebook personal computer, a tablet terminal, or a smart phone, or the like can be used. Furthermore, in the structure shown in Figure 1 , the digital camera 100 and the image processing device 110 are configured separately from each other, but are not limited thereto. They can be configured integrally, for example, that is, can be configured such that the functions of the image processing device 110 are incorporated in the digital camera 100.

[0020] The HMD 120 can be communicably coupled to the image processing apparatus 110. Thus, the user 130 can enjoy the double fisheye image obtained by performing image processing in the image processing apparatus 110 while wearing the HMD 120 on the head. In this case, the image processing apparatus 110 converts the double fisheye image into a format suitable for streaming, and transmits the double fisheye image directly to the HMD 120, or distributes the double fisheye image to one or more HMDs 120 of one or more users 130 via the cloud 125. Note that the communication method between the HMD 120 and the image processing apparatus 110 is not particularly limited, and can be, for example, a wireless communication method or a wired communication method.

[0021] Figure 2 is a block diagram illustrating an example of a hardware structure of an image processing apparatus. As Figure 2 indicated, the image processing apparatus 110 includes a controller 205, a ROM 210, a RAM 215, an external storage 220, an operation unit 225, a display unit 230, a communication unit 235, and an external I / F 240, which are communicably connected to each other via a system bus 245.

[0022] The controller 205 is a computer including, for example, a CPU and controls the entire image processing apparatus 110. The ROM 210 stores programs, parameters, and the like. The programs include, for example, a program that causes the controller 205 to execute the steps (control method of an image processing apparatus) described later. The RAM 215 temporarily stores programs, data, and the like provided from an external device.

[0023] The external storage 220 stores various programs, data, and the like. The external storage 220 is not particularly limited, and can be, for example, a hard disk or a flash memory fixedly installed in the image processing apparatus 110, or an FD or an optical disk such as a CD, which is detachably installed in the image processing apparatus 110. The external storage 220 can be a magnetic card, an optical card, an IC card, or a memory card, or the like. The operation unit 225 includes a button or a touch panel that accepts a user operation and inputs data, or the like.

[0024] The display unit 230 displays, for example, data stored in the image processing apparatus 110. The display unit 230 is not particularly limited, and can be, for example, a liquid crystal display. The communication unit 235 is connected to the Internet or the like. The external I / F 240 can receive the double fisheye image from the digital camera 100. The external I / F 240 can transmit data stored in the image processing apparatus 110 to an external device.

[0025] Figure 3 is a flowchart illustrating a process performed by an image processing apparatus. Figure 4A is a diagram for describing the processes in steps S305 and S309 in the flowchart illustrated in Figure 3 .Figure 4B is for describing Figure 3 a diagram of the processing in steps S308 and S310 in the flowchart shown in FIG. 8. As Figure 3 indicated in step S300, the controller 205 of the image processing device 110 determines whether the HDMI signal output from the digital camera 100 is received with the external I / F 240.

[0026] As a result of the determination in step S300, when the controller 205 determines that the HDMI signal is received, the processing proceeds to step S301. On the other hand, as a result of the determination in step S300, when the controller 205 determines that the HDMI signal is not received, the processing waits in step S300.

[0027] In step S301, the controller 205 determines whether the output of the HDMI signal from the digital camera 100 is stopped. As a result of the determination in step S301, when the controller 205 determines that the output of the HDMI signal is stopped, the processing ends. On the other hand, as a result of the determination in step S301, when the controller 205 determines that the output of the HDMI signal is not stopped, the processing proceeds to step S302.

[0028] In step S302, the controller 205 reads the image data included in the HDMI signal received in step S300. Next, when the image data includes the data of the circular fisheye image, the controller 205 can obtain the circular fisheye image. In this way, in the present embodiment, the controller 205 functions as an obtaining unit that obtains the circular fisheye image. In the image processing device 110, a component that functions as the obtaining unit can be provided separately from the controller 205.

[0029] In step S303, the controller 205 determines whether the image data read in step S302 includes the left-eye image and the right-eye image as the above-described two circular fisheye images. In this way, in the present embodiment, the controller 205 functions as a determination unit that determines whether the left-eye image and the right-eye image are included. In the image processing device 110, a component that functions as the determination unit can be provided separately from the controller 205. For example, the determination in step S303 is made in accordance with a known method using the luminance of the image. Then, as a result of the determination in step S303, when the controller 205 determines that the left-eye image and the right-eye image are included, the processing proceeds to step S304. On the other hand, as a result of the determination in step S303, when the controller 205 determines that the left-eye image and the right-eye image are not included, the processing returns to step S301 and sequentially executes the subsequent steps. In this case, the image processing described later is omitted.

[0030] In step S304, the controller 205 determines whether the image data read in step S302 is data of an even frame. As a result of the determination in step S304, when the controller 205 determines that the frame is an even frame, the process proceeds to step S305. On the other hand, as a result of the determination in step S304, when the controller 205 determines that the frame is not an even frame (i.e., the frame is an odd frame), the process proceeds to step S306.

[0031] In step S305, the controller 205 performs image processing (image processing step) on the left eye image (one image) among the two circular fisheye images including the left eye image and the right eye image that are determined to have been acquired in step S303, to reduce distortion, and acquires a left eye image with reduced distortion. In this way, in the present embodiment, the controller 205 functions as an image processing unit that performs image processing to reduce distortion of the circular fisheye image. In the image processing device 110, a component that functions as the image processing unit can be provided separately from the controller 205.

[0032] Figure 4A The circular fisheye image 400 in FIG. 4 is an image captured by the digital camera 100 to which the lens unit 105 is attached. The circular fisheye image 400 includes a left eye image 401L formed by the left lens of the lens unit 105 and a right eye image 401R formed by the right lens of the lens unit 105. Then, in step S305, as the image processing on the left eye image 401L, image conversion processing based on equirectangular projection is performed. As a result, the left eye image 401L is converted into a left eye image 401L' with reduced distortion.

[0033] In step S305, preferably, before the image processing, disparity correction processing that corrects disparity between the left eye image and the right eye image is performed on the left eye image and the right eye image. This allows the user to view the image more stereoscopically on the HMD 120. The disparity correction processing is performed taking into account the influence of the deviation between the optical axes of the left lens and the right lens of the lens unit 105, based on individual differences such as assembly errors of the lens unit 105, and meta-information such as the posture of the digital camera 100 at the time of imaging and the temperature environment at the time of imaging. In the disparity correction processing, in a case where only one of the left eye image and the right eye image is selected, the disparity correction of the one image can be performed with the other image fixed.

[0034] In step S306, similarly to step S305 (image processing step), the controller 205 performs image processing on the right eye image (one image) among the images including the left eye image and the right eye image that are determined in step S303 to reduce distortion.

[0035] In step S307, the controller 205 determines whether the image processing in step S305 or S306 has been executed and completed within a predetermined time period. In this embodiment, as an example, 60 frames of images are received per second in step S300. In this case, the image processing in step S305 or S306 needs to be executed within approximately 16 milliseconds. In order to perform image processing for the left-eye and right-eye images within 16 milliseconds under this condition, the image processing for each image needs to be completed within 8 milliseconds. Therefore, when the image processing for one of the left-eye and right-eye images is completed within 8 milliseconds, it is determined that the image processing for the other image will also be completed within the remaining 8 milliseconds (=16 milliseconds - 8 milliseconds).

[0036] Then, as a result of the determination in step S307, when the controller 205 determines that the image processing is completed within a predetermined time period (8 milliseconds), the processing proceeds to step S308. On the other hand, as a result of the determination in step S307, when the controller 205 determines that the image processing is not completed within the predetermined time period (8 milliseconds), the image processing for another image will not be completed in time, and the processing proceeds to step S309. Although in this embodiment, the determination in step S307 is configured to determine whether the processing for the monocular image is completed within the predetermined time at the point when the processing for the monocular image is completed, this is not limited to this. For example, the determination in step S307 could be configured to determine whether the processing for the binocular image is completed when a predetermined time has elapsed since the start of the processing.

[0037] In step S308, the controller 205 performs image processing on another image to reduce distortion (image processing step). That is, as processing proceeds through step S305, the controller 205 performs image processing on the right-eye image, and as processing proceeds through step S306, it performs image processing on the left-eye image. The image processing for each image is similar to the image processing in step S305.

[0038] and Figure 4A Similarly, Figure 4B The circular fisheye image 400 is captured by a digital camera 100 with a lens unit 105 attached. The circular fisheye image 400 includes a left-eye image 401L and a right-eye image 401R. Then, in step S305, image processing is performed on the left-eye image 401L. Therefore, the left-eye image 401L is converted into a distortion-reduced left-eye image 401L'. In addition, in step S308, as image processing for the right-eye image 401R, image transformation processing based on equal rectangular projection is performed. As a result, the right-eye image 401R is converted into a distortion-reduced right-eye image 401R'.

[0039] In step S309, the controller 205 outputs the left eye image (one image) that has been processed in step S305 or the right eye image (one image) that has been processed in step S306 to the HMD 120 via the communication unit 235 (output unit). Figure 4A This shows the state of the output left eye image 401L'.

[0040] In step S310, the controller 205 outputs the image (one image) that was processed in step S305 or step S306 and the image (another image) that was processed in step S308 to the HMD120 via the communication unit 235. Figure 4B This shows the output states of the left-eye image 401L' and the right-eye image 401R'.

[0041] Figure 5 This is a flowchart illustrating the processes performed by the HMD. Figure 6A It is used to describe Figure 5 The flowchart shown illustrates the processing of step S503. Figure 6B It is used to describe Figure 5 The flowchart shown illustrates the processing of step S505. Figure 5 As shown, in step S500, the controller of HMD 120 (hereinafter referred to as the "HMD controller") determines whether an image signal output from the communication unit 235 of image processing device 110 has been received. The image signal includes step S309 (see...). Figure 3 The image output in step S310 (see) Figure 3 The image output in ).

[0042] Then, as a result of the judgment in step S500, when the HMD controller determines that an image signal has been received, the process proceeds to step S501. On the other hand, as a result of the judgment in step S500, when the HMD controller determines that no image signal has been received, the process waits in step S500.

[0043] In step S501, the HMD controller determines whether the output of the image signal from the image processing device 110 has stopped. As a result of the determination in step S501, if the HMD controller determines that the output of the image signal has stopped, the process ends. On the other hand, as a result of the determination in step S501, if the HMD controller determines that the output of the image signal has not stopped, the process proceeds to step S502.

[0044] In step S502, the HMD controller determines whether the circular fisheye image included in the image signal received in step S500 is a left-eye image or a right-eye image, i.e., a monocular image (one image). As described above, in this embodiment, the HMD controller is used as a determination unit to determine whether the circular fisheye image obtained by the HMD controller (acquisition unit) is a monocular image. In the HMD 120, the component used as the determination unit can be set separately from the HMD controller. Then, as the determination result in step S502, when the HMD controller determines that a monocular image has been received, the process proceeds to step S503. On the other hand, as the determination result in step S502, when the HMD controller determines that the received image is not a monocular image, the process proceeds to step S504.

[0045] In step S503, the HMD controller updates the monocular image currently displayed on the HMD 120 using the monocular image included in the image signal received in step S500 and which has undergone image processing. Note that this update is performed only on the region corresponding to the monocular image included in the image signal received in step S500. Regions not corresponding to the monocular image continue to be displayed as in the previous frame. In this way, in this embodiment, the HMD controller acts as an update unit for updating the image. In the HMD 120, the component used as the update unit can be disposed separately from the HMD controller.

[0046] Figure 6A The circular fisheye image 600 in the video is the image (video) displayed on the HMD 120 in the Nth frame. For example, when only the image-processed left-eye image is sent from the image processing device 110 in the (N+1)th frame, the HMD 120 receives only the left-eye image and updates the display to reflect the relevant left-eye image. On the other hand, the right-eye image of the Nth frame continues to be displayed as the right-eye image in the (N+1)th frame. Similarly, when only the image-processed right-eye image is sent from the image processing device 110 in the (N+2)th frame, the HMD 120 receives only the right-eye image and updates the display to reflect the relevant right-eye image. On the other hand, the left-eye image of the (N+1)th frame continues to be displayed as the left-eye image in the (N+2)th frame.

[0047] In step S504, the HMD controller determines whether the circular fisheye image included in the image signal received in step S500 includes both a left-eye image and a right-eye image, i.e., a binocular image. As the result of the determination in step S504, if the HMD controller determines that a binocular image has been received, the process proceeds to step S505. Conversely, if the HMD controller determines that no binocular image has been received, the process returns to step S501, and subsequent steps are executed sequentially.

[0048] In step S505, the HMD controller updates the binocular images currently displayed on the HMD 120 to the binocular images included in the image signal received in step S500 and having undergone image processing. Figure 6A Similarly, Figure 6B The circular fisheye image 600 in the Nth frame is the image displayed on the HMD 120. For example, when image-processed binocular images are sent from the image processing device 110 in the (N+1)th frame, the HMD 120 receives the binocular images and updates the display to reflect the relevant binocular images. Although the update processing in steps S503 and S505 is performed by the HMD controller in this embodiment, it is not limited to this. For example, the controller 205 (update control unit) of the image processing device 110 can perform the update processing in steps S503 and S505. This can reduce the processing load on the HMD 120.

[0049] As described above, in the image processing system 1000, the fisheye image (moving image) captured by the digital camera 100 can be streamed live to the HMD 120 in real time by the image processing device 110. At this time, the image processing device 110 can perform image processing for the left-eye and right-eye images based on the frame rate during streaming. Furthermore, the HMD 120 can update the currently displayed image to the image that has already undergone image processing in the image processing device 110, i.e., the processed image. Therefore, the user can observe a smooth live streaming of the moving image on the HMD 120.

[0050] According to this disclosure, image processing can be performed based on the frame rate during streaming.

[0051] Other embodiments

[0052] Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs / instructions) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU), microprocessor unit (MPU) of the system or device) reads and executes the computer program / instructions.

[0053] While this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the appended claims is to be interpreted in the broadest sense to include all such modifications and equivalent structures and functions.

[0054] This application claims the benefit of Japanese Patent Application 2024-166161, filed on September 25, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing apparatus communicatively connected to a display device displaying a left-eye image viewable by the left eye and a right-eye image viewable by the right eye, the image processing apparatus comprising: Memory devices that store instruction sets; as well as At least one processor that executes the instruction set to: Obtain a fisheye image, which is a moving image captured using a fisheye lens; In the case where the fisheye image includes the left eye image and the right eye image, image processing is performed on one of the left eye image and the right eye image to reduce distortion; as well as If the image processing for one image is completed within a predetermined time period, the image processing is performed on another image.

2. The image processing device according to claim 1, wherein The at least one processor executes instructions from the memory device to: Determine whether the obtained fisheye image includes the left eye image and the right eye image; If, as a result of the determination, the fisheye image is determined to include the left eye image and the right eye image, then the image processing is performed on the one image. If the image processing for one image is completed within the predetermined time period, the image processing is performed on the other image; If it is determined that the fisheye image does not include the left eye image and the right eye image, the image processing is omitted.

3. The image processing device according to claim 2, wherein The at least one processor executes instructions in the memory device to perform a process to correct the parallax between the left-eye image and the right-eye image if, as a result of the determination, the fisheye image is determined to include the left-eye image and the right-eye image.

4. The image processing device according to any one of claims 1 to 3, wherein The at least one processor executes instructions in the memory device to output the image that has undergone the image processing and the image that has undergone the image processing to the display device, provided that the image processing has already been performed on the other image.

5. The image processing device according to claim 4, wherein The at least one processor executes instructions in the memory device to output the image that has undergone the image processing to the display device if the image processing for the image is not completed within the predetermined time period.

6. The image processing device according to any one of claims 1 to 3, 5, wherein The at least one processor executes instructions in the memory device to perform the image processing on the left eye image when the fisheye image is an even number of frames, and to perform the image processing on the right eye image when the fisheye image is an odd number of frames.

7. The image processing apparatus according to any one of claims 1 to 3, and 5, wherein The at least one processor executes instructions in the memory device to allow the execution of a transformation process based on isorectangular projection as the image processing for the fisheye image.

8. The image processing device according to any one of claims 1 to 3, 5, wherein, The fisheye image is a circular fisheye image.

9. The image processing device according to any one of claims 1 to 3, 5, wherein, The at least one processor executes instructions from the memory device to: Allows control over updating the fisheye image displayed on the display device; as well as in a case where the fisheye image includes the one image subjected to the image processing, updating one image currently being displayed to the one image subjected to the image processing.

10. A display device communicably connectable to an image processing apparatus, the display device comprising: a memory device storing a set of instructions; and at least one processor that executes the set of instructions to: obtain a fisheye image from the image processing apparatus, the fisheye image being a moving image captured using a fisheye lens; display the obtained fisheye image; in a case where the fisheye image includes one image subjected to image processing to reduce distortion, update one image currently being displayed to the one image subjected to the image processing.

11. The display device of claim 10, wherein, the at least one processor executes the instructions in the memory device to: determine whether the obtained fisheye image includes the one image subjected to the image processing; and in a case where, as a result of the determination, the fisheye image includes the one image subjected to the image processing, update one image currently being displayed to the one image subjected to the image processing.

12. The display device according to claim 10 or 11, wherein, the display device is a head-mounted display.

13. A control method of an image processing apparatus communicably connectable to a display device that displays a left-eye viewable image and a right-eye viewable image, the control method comprising: obtaining a fisheye image, the fisheye image being a moving image captured using a fisheye lens; in a case where the fisheye image includes the left-eye viewable image and the right-eye viewable image, performing image processing on one image among the left-eye viewable image and the right-eye viewable image to reduce distortion; and in a case where the image processing on the one image is completed within a predetermined period of time, performing the image processing on another image.

14. A non-transitory computer-readable storage medium storing a program that causes a computer to execute the control method according to claim 13.

15. A computer program product including a program that causes a computer to execute the control method according to claim 13. ​

Images