Composite image generation apparatus and method, and non-transitory computer readable storage medium

Abstract

The invention provides a composite image generation device and method and a non-transitory computer readable storage medium thereof. The device determines an eye gaze position corresponding to a user. The device determines a region of interest corresponding to a plurality of instant images based on the eye gaze position, and the plurality of instant images each correspond to an exposure value and a resolution. The apparatus generates a composite image based on the region of interest and a non-region of interest corresponding to the plurality of live images, and the region of interest and the non-region of interest of the composite image are generated based on the plurality of live images corresponding to different resolutions. The device transmits the composite image to a display device for an instant display operation. The synthetic image generation technology provided by the invention solves the problem that the prior art cannot be applied to instant display.

Description

Technical Field

[0001] The present invention relates to a synthetic image generating device, method, and non-transitory computer-readable storage medium. Specifically, the present invention relates to a synthetic image generating device, method, and non-transitory computer-readable storage medium for improving the efficiency of generating synthetic images. Background Art

[0002] In recent years, various technologies related to virtual reality have developed rapidly, and various related technologies and applications have been proposed one after another.

[0003] In the prior art, when performing an interactive operation, a head-mounted device can capture an instant image of a physical space through a camera disposed in the environment or on the device and display it on a display screen (e.g., through an optical see-through or video see-through operation).

[0004] However, since there may be extremely bright and extremely dark picture areas in the instant image (e.g., a window strongly irradiated by sunlight and a corner of a room), the content in the image cannot be clearly presented (e.g., the phenomenon of overexposure or underexposure of the image), resulting in a poor user experience.

[0005] In addition, even though the prior art can generate an image with a larger brightness range by synthesizing multiple instant images with different exposure ranges. However, since processing an instant image containing a large amount of image details requires a large amount of computation and time costs, the frames per second (FPS) of the displayed image is reduced. Since in a head-mounted device, an excessively low frames per second will cause dizziness to the user, the existing synthesis method still cannot be used in an instant display device.

[0006] In view of this, how to provide a synthetic image technology for improving the efficiency of generating synthetic images is an urgent goal that the industry needs to strive for. Summary of the Invention

[0007] An object of the present invention is to provide a synthetic image generating device. The synthetic image generating device includes a transceiver interface and a processor, and the processor is electrically connected to the transceiver interface. The processor determines a gaze position of an eye corresponding to a user. The processor determines a region of interest corresponding to a plurality of live images based on the eye gaze position, wherein each of the plurality of live images corresponds to an exposure value and a resolution. The processor generates a synthetic image based on the region of interest and a non-interest region corresponding to the plurality of live images, wherein the region of interest and the non-interest region of the synthetic image are generated based on the plurality of live images corresponding to different resolutions. The processor transmits the synthetic image to a display device for an immediate display operation.

[0008] Another object of the present invention is to provide a synthetic image generating method, which is used for an electronic device. The synthetic image generating method includes the following steps: determining a gaze position of an eye corresponding to a user; determining a region of interest corresponding to a plurality of live images based on the eye gaze position, wherein each of the plurality of live images corresponds to an exposure value and a resolution; generating a synthetic image based on the region of interest and a non-interest region corresponding to the plurality of live images, wherein the region of interest and the non-interest region of the synthetic image are generated based on the plurality of live images corresponding to different resolutions; and transmitting the synthetic image to a display device for an immediate display operation.

[0009] Another object of the present invention is to provide a non-transitory computer-readable storage medium, which stores a computer program. The computer program includes a plurality of program instructions. After the computer program is loaded into an electronic device, it executes a synthetic image generating method. The synthetic image generating method includes the following steps: determining a gaze position of an eye corresponding to a user; determining a region of interest corresponding to a plurality of live images based on the eye gaze position, wherein each of the plurality of live images corresponds to an exposure value and a resolution; generating a synthetic image based on the region of interest and a non-interest region corresponding to the plurality of live images, wherein the region of interest and the non-interest region of the synthetic image are generated based on the plurality of live images corresponding to different resolutions; and transmitting the synthetic image to a display device for an immediate display operation.

[0010] In an embodiment of the present invention, generating the synthetic image further includes the following operations: generating a plurality of first regional pixel values corresponding to the region of interest based on the plurality of real-time images corresponding to a first resolution; generating a plurality of second regional pixel values corresponding to the non-region of interest based on the plurality of real-time images corresponding to a second resolution; and synthesizing the plurality of first regional pixel values and the plurality of second regional pixel values to generate the synthetic image, wherein the first resolution is higher than the second resolution.

[0011] In an embodiment of the present invention, the plurality of real-time images corresponding to the second resolution are generated by the following operations: calculating a brightness value of a photometric image at the eye fixation position; selecting at least one second real-time image from the plurality of real-time images based on the brightness value of the photometric image at the eye fixation position and the brightness values of the plurality of real-time images corresponding to the region of interest; and performing a downsampling operation on the at least one second real-time image to generate the plurality of real-time images corresponding to the second resolution.

[0012] In an embodiment of the present invention, determining the eye fixation position of the user further includes the following operations: determining the eye fixation position of the user based on eye tracking information of the user in a photometric image.

[0013] In an embodiment of the present invention, the plurality of real-time images are generated by the following operations: calculating a brightness value of the photometric image at the eye fixation position; and determining the exposure value and the resolution corresponding to each of the plurality of real-time images based on the brightness value of the photometric image at the eye fixation position, wherein the plurality of real-time images are generated by at least one image capturing device based on the exposure value and the resolution corresponding to each of the plurality of real-time images.

[0014] In an embodiment of the present invention, determining the region of interest further includes the following operations: calculating a brightness value of the photometric image at the eye fixation position; generating a plurality of target pixel positions in the photometric image corresponding to the brightness value; and determining the region of interest based on the plurality of target pixel positions.

[0015] In an embodiment of the present invention, determining the region of interest further includes the following operations: identifying a target object corresponding to the eye fixation position in the photometric image; generating a plurality of target pixel positions in the photometric image corresponding to the target object; and determining the region of interest based on the plurality of target pixel positions.

[0016] In an embodiment of the present invention, wherein the plurality of instant images are generated by a single image capturing device, the single image capturing device corresponding to a plurality of first exposure parameters and a plurality of first resolution parameters, and the plurality of instant images are generated by the following operations: the single image capturing device generates the plurality of instant images based on the plurality of first exposure parameters and the plurality of first resolution parameters.

[0017] In an embodiment of the present invention, wherein the plurality of instant images are generated by a plurality of image capturing devices, the plurality of image capturing devices each corresponding to a second exposure parameter and a second resolution parameter, and the plurality of instant images are generated by the following operations: the plurality of image capturing devices generate the plurality of instant images based on the second exposure parameter and the second resolution parameter corresponding to each of the plurality of image capturing devices.

[0018] In an embodiment of the present invention, wherein the display device is a head-mounted display device, and the head-mounted display device is worn by the user.

[0019] In an embodiment of the present invention, wherein generating the composite image further includes the following steps: generating a plurality of first regional pixel values corresponding to the region of interest based on the plurality of instant images corresponding to a first resolution; generating a plurality of second regional pixel values corresponding to the non-region of interest based on the plurality of instant images corresponding to a second resolution; and synthesizing the plurality of first regional pixel values and the plurality of second regional pixel values to generate the composite image, wherein the first resolution is higher than the second resolution.

[0020] In an embodiment of the present invention, wherein the plurality of instant images corresponding to the second resolution are generated by the following steps: calculating a luminance value of a photometric image at the eye fixation position; selecting at least one second instant image from the plurality of instant images based on the luminance value of the photometric image at the eye fixation position and the luminance values of the plurality of instant images corresponding to the region of interest; and performing a downsampling operation on the at least one second instant image to generate the plurality of instant images corresponding to the second resolution.

[0021] In an embodiment of the present invention, wherein determining the eye fixation position of the user further includes the following steps: determining the eye fixation position of the user based on the eye tracking information of the user in a photometric image.

[0022] In an embodiment of the present invention, the plurality of instant images are generated by the following steps: calculating a brightness value of the photometric image at the eye fixation position; and determining the exposure value and the resolution corresponding to each of the plurality of instant images based on the brightness value of the photometric image at the eye fixation position, wherein the plurality of instant images are generated by at least one image capturing device based on the exposure value and the resolution corresponding to each of the plurality of instant images.

[0023] In an embodiment of the present invention, determining the region of interest further includes the following steps: calculating a brightness value of the photometric image at the eye fixation position; generating a plurality of target pixel positions corresponding to the brightness value in the photometric image; and determining the region of interest based on the plurality of target pixel positions.

[0024] In an embodiment of the present invention, determining the region of interest further includes the following steps: identifying a target object corresponding to the eye fixation position in the photometric image; generating a plurality of target pixel positions corresponding to the target object in the photometric image; and determining the region of interest based on the plurality of target pixel positions.

[0025] In an embodiment of the present invention, the plurality of instant images are generated by a single image capturing device, the single image capturing device corresponds to a plurality of first exposure parameters and a plurality of first resolution parameters, and the plurality of instant images are generated by the following steps: generating the plurality of instant images by the single image capturing device based on the plurality of first exposure parameters and the plurality of first resolution parameters.

[0026] In an embodiment of the present invention, the plurality of instant images are generated by a plurality of image capturing devices, each of the plurality of image capturing devices corresponds to a second exposure parameter and a second resolution parameter, and the plurality of instant images are generated by the following steps: generating the plurality of instant images by the plurality of image capturing devices based on the second exposure parameter and the second resolution parameter corresponding to each of the plurality of image capturing devices.

[0027] The synthetic image generation technology provided by the present disclosure (at least including devices, methods, and their non-transitory computer-readable storage media) performs corresponding photometric operations by analyzing the eye fixation position of the user, and determines the components of the synthetic image based on the brightness value of the eye fixation position. The synthetic image generation technology provided by the present disclosure provides for composing important parts of the synthetic image with instant images of higher resolution, and composing less important parts of the synthetic image with instant images of lower resolution, improving the efficiency of the synthetic image. Since the synthetic image generation technology provided by the present disclosure solves the problem that the prior art cannot be applied to instant display, the service experience of the user is improved.

[0028] The following describes in detail the technology and implementation of the present invention in conjunction with the accompanying drawings, so that those with ordinary knowledge in the technical field to which the present invention pertains can understand the technical features of the claimed invention. Description of the Drawings

[0029] Figure 1 Schematic diagram showing a synthetic image generation device of the first embodiment;

[0030] Figure 2 Schematic diagram showing a photometric image of certain embodiments;

[0031] Figure 3 Schematic diagram showing an instantaneous image of certain embodiments;

[0032] Figure 4 Schematic diagram showing an instantaneous image of certain embodiments; and

[0033] Figure 5 Partial flowchart showing a synthetic image generation method of the second embodiment.

[0034] Symbol Explanation:

[0035] 1: Synthetic image generation device

[0036] 11: Transceiver interface

[0037] 13: Processor

[0038] 200: Photometric image

[0039] ROI: Region of interest

[0040] RONI: Region of non - interest

[0041] FL: Fluorescent lamp

[0042] IM301, IM302: Instantaneous images

[0043] IM401, IM402, IM403: Instantaneous images

[0044] 500: Synthetic image generation method

[0045] S501, S503, S505, S507: Steps Detailed Implementation Modes

[0046] The following will explain a synthetic image generation device, method, and its non-transitory computer-readable storage medium provided by the present invention through embodiments. However, the multiple embodiments are not used to limit that the present invention must be implemented in any environment, application, or manner as described in the multiple embodiments. Therefore, the description of the embodiments is only for the purpose of explaining the present invention and not for limiting the scope of the present invention. It should be understood that in the following embodiments and drawings, elements not directly related to the present invention have been omitted and not shown, and the dimensions of each element and the dimensional ratios between elements are only examples and not used to limit the scope of the present invention.

[0047] The first embodiment of the present invention is a synthetic image generation device 1, and its schematic architecture diagram is shown in Figure 1 . In this embodiment, the synthetic image generation device 1 includes a transceiver interface 11 and a processor 13, and the processor 13 is electrically connected to the transceiver interface 11.

[0048] It should be noted that the processor 13 can be various processing units, a central processing unit (CPU), a microprocessor, or other computing devices known to those with ordinary knowledge in the technical field to which the present disclosure belongs. The transceiver interface 11 is an interface that can receive and transmit data or other interfaces that can receive and transmit data known to those with ordinary knowledge in the technical field of this case.

[0049] In some embodiments, the synthetic image generation device 1 can be communicatively connected to a display device (for example: a head mounted display (HMD)) to transmit the generated synthetic image to the display device for immediate display.

[0050] In some embodiments, the synthetic image generation device 1 can be disposed in other devices or combined with a device having computing capabilities (for example: sharing a processor with a device). For example, the synthetic image generation device 1 can be disposed in a head mounted display, the processor 13 can be a built-in processor in the head mounted display, the transceiver interface 11 can be a built-in transceiver interface in the head mounted display, and the synthetic image generation device 1 can transmit the generated synthetic image to the display device in the head mounted display for immediate display.

[0051] First, in this embodiment, the processor 13 in the synthetic image generation device 1 determines the eye fixation position corresponding to the user (for example: a user using a head mounted display). Specifically, the processor 13 can determine the eye fixation position of the user based on the eye tracking information of the user in a photometric image.

[0052] In some embodiments, the eye tracking information can be generated by analyzing the positions where the user's both eyes fixate on the screen (e.g., eye tracking technology).

[0053] In some embodiments, the processor 13 can perform photometry on the environment in advance (e.g., by using a pre-generated photometry image or using one of multiple real-time images as the photometry image), and determine the position where the user's eyes fixate.

[0054] It should be noted that in some embodiments, the processor 13 can first perform photometry on the eye fixation position, and then generate corresponding multiple real-time images based on the photometry result.

[0055] Next, in this embodiment, the processor 13 determines a region of interest corresponding to multiple real-time images based on the eye fixation position, where each of the multiple real-time images corresponds to an exposure value and a resolution.

[0056] In some embodiments, the processor 13 can use the target pixel positions with similar brightness values as the region of interest. Specifically, the processor 13 calculates a brightness value of the photometry image at the eye fixation position. Then, the processor 13 generates multiple target pixel positions in the photometry image corresponding to the brightness value. Finally, the processor 13 determines the region of interest based on the multiple target pixel positions.

[0057] For easy understanding, please refer to Figure 2 the schematic diagram 200 of the real-time image. As Figure 2 shown, the processor 13 determines that the eye fixation position EGP corresponding to the user is located at the position of the window. In this example, since the window part has similar brightness values (i.e., falling within a threshold range), the processor 13 uses the window area with similar brightness values as the region of interest ROI.

[0058] In some examples, when the measured brightness value of the area of the fluorescent lamp FL is close to that of the window area, the processor 13 can also add the area of the fluorescent lamp FL to the region of interest ROI. Therefore, the region of interest ROI can simultaneously include the areas of the window and the fluorescent lamp FL.

[0059] In some embodiments, the processor 13 can use the object that the eye fixation position focuses on as the region of interest based on the operation of identifying the object. Specifically, the processor 13 identifies a target object corresponding to the eye fixation position in the photometry image. Then, the processor 13 generates multiple target pixel positions in the photometry image corresponding to the target object. Finally, the processor 13 determines the region of interest based on the multiple target pixel positions.

[0060] For example, as Figure 2As shown, the processor 13 identifies that the object of interest of the user's eye gaze position is a window, and the processor 13 calculates the pixel area where the window appears in the image as the region of interest ROI.

[0061] Next, in the present embodiment, the processor 13 generates a composite image based on the region of interest and a non - region of interest corresponding to the plurality of live images, wherein the region of interest and the non - region of interest of the composite image are generated based on the plurality of live images corresponding to different resolutions.

[0062] In some embodiments, the pixel positions that do not belong to the region of interest are regarded as the non - region of interest. Specifically, after the processor 13 determines the region of interest corresponding to the live image, it generates the non - region of interest corresponding to the remaining other regions.

[0063] In some embodiments, since the region of interest is the region that the user pays more attention to (i.e., the region of interest where the user's eye gaze position is located), the processor 13 will synthesize the region of interest of the composite image with the live image having a higher resolution, and synthesize the non - region of interest of the composite image with the live image having a lower resolution, reducing the resource cost of the operation. Specifically, the processor 13 generates a plurality of first region pixel values corresponding to the region of interest based on the plurality of live images corresponding to a first resolution. Then, the processor 13 generates a plurality of second region pixel values corresponding to the non - region of interest based on the plurality of live images corresponding to a second resolution. Finally, the processor 13 synthesizes the plurality of first region pixel values and the plurality of second region pixel values to generate the composite image, wherein the first resolution is higher than the second resolution.

[0064] In some embodiments, the processor 13 can extract the pixel values corresponding to each pixel position in the region of interest from the plurality of live images corresponding to the first resolution as the plurality of first region pixel values (i.e., the pixel values of all pixel positions located in the region of interest). In addition, the processor 13 can extract the pixel values corresponding to each pixel position in the non - region of interest from the plurality of live images corresponding to the second resolution as the plurality of second region pixel values (i.e., the pixel values of all pixel positions located in the non - region of interest).

[0065] In some embodiments, the processor 13 can determine the exposure value and the resolution corresponding to each of the plurality of live images through a brightness value of the eye gaze position. By increasing the resolution of the live images with similar brightness values and decreasing the resolution of other live images, the cost consumption during calculation is reduced.

[0066] Specifically, the processor 13 calculates a brightness value of the photometric image at the eye fixation position. Then, based on the brightness value of the photometric image at the eye fixation position, the processor 13 determines the exposure value and the resolution corresponding to each of the multiple instantaneous images. It should be noted that the multiple instantaneous images are generated by at least one image capturing device based on the exposure value and the resolution corresponding to each of the multiple instantaneous images.

[0067] In some embodiments, the instantaneous image among the multiple instantaneous images that is close to the brightness value at the eye fixation position corresponds to the highest resolution.

[0068] In some embodiments, the processor 13 can also actively reduce the resolution corresponding to some of the instantaneous images (i.e., the instantaneous images corresponding to the non - region of interest components) to reduce the cost consumption during calculation.

[0069] It should be noted that the resolution reduction of the present disclosure can perform the image capturing operation with lower parameter settings when the image capturing device takes pictures or actively reduce the resolution corresponding to the instantaneous images in a post - production manner.

[0070] In some embodiments, the processor 13 can actively perform a resolution reduction operation on the instantaneous images that are irrelevant to synthesizing the region of interest to reduce the resource cost of the operation. Specifically, the processor 13 calculates a brightness value of a photometric image at the eye fixation position. Then, based on the brightness value of the photometric image at the eye fixation position and the brightness values of the multiple instantaneous images corresponding to the region of interest, the processor 13 selects at least one second instantaneous image from the multiple instantaneous images. Finally, the processor 13 performs a resolution reduction operation on the at least one second instantaneous image to generate the multiple instantaneous images corresponding to the second resolution.

[0071] Finally, in this embodiment, the processor 13 transmits the synthesized image to a display device for an instant display operation.

[0072] In some embodiments, the display device is a head - mounted display, and the head - mounted display is worn by the user.

[0073] For ease of understanding, taking the example where the processor 13 generates a synthesized image based on two instantaneous images, please refer to Figure 3 the schematic diagrams of the instantaneous images IM301 and IM302. In this example, the instantaneous image IM301 has a lower exposure value (i.e., underexposed) and a higher resolution, and the instantaneous image IM302 has a higher exposure value (i.e., overexposed) and a lower resolution.

[0074] It should be noted that since the exposure value of the instant image IM301 is low, the details of the window area with higher brightness are clearer. In addition, since the exposure value of the instant image IM302 is high, the pixel details of the window area with higher brightness will contain less pixel details due to overexposure.

[0075] In this example, since the region of interest ROI corresponds to the window area with higher brightness, the processor 13 extracts multiple regional pixel values of the region of interest ROI of the instant image IM301 (i.e., the bright details are clearer) as part of the composite image (i.e., the region of interest ROI of the composite image). In addition, the processor 13 extracts multiple regional pixel values of the region of non-interest RONI of the instant image IM302 (i.e., the bright details are blurrier) as another part of the composite image (i.e., the region of non-interest RONI of the composite image).

[0076] In addition, taking the generation of a composite image by the processor 13 based on three instant images as an example, please refer to Figure 4 the instant image schematic diagrams IM401, IM402, and IM403. In this example, the instant image IM401 has a low exposure value (i.e., underexposed) and a high resolution, the instant image IM402 has a normal exposure value and a low resolution, and the instant image IM403 has a high exposure value (i.e., overexposed) and a low resolution.

[0077] It should be noted that since the exposure value of the instant image IM401 is low, the details of the window area with higher brightness are clearer. In addition, since the exposure values of the instant images IM402 and IM403 are high, the pixel details of the window area with higher brightness will contain less pixel details due to overexposure.

[0078] In this example, since the region of interest ROI corresponds to the window area with higher brightness, the processor 13 extracts multiple regional pixel values of the region of interest ROI of the instant image IM401 (i.e., the bright details are clearer) as part of the composite image (i.e., the region of interest ROI of the composite image). In addition, the processor 13 extracts multiple regional pixel values of the region of non-interest RONI of the instant images IM402 and IM403 (i.e., the bright details are blurrier) as another part of the composite image (i.e., the region of non-interest RONI of the composite image).

[0079] It should be noted that the instant images required for the synthetic image of the present disclosure are at least two instant images. The present disclosure does not limit the number of instant images used for synthesizing the image. Those with ordinary knowledge in the art should be able to understand the implementation manners when there are more instant images based on the description content of the present disclosure, so no more elaboration will be made. Additionally, when the processor 13 synthesizes the Region of Non-Interest (RONI) through multiple instant images, the RONI can be synthesized by means of weight ratios and the like.

[0080] It should be noted that in the present disclosure, the multiple instant images can be generated by a single image capturing device or by multiple image capturing devices (for example: cameras with better and worse resolutions).

[0081] In some embodiments, a single image capturing device can continuously capture instant images with different resolutions by setting different exposure times (for example: the single image capturing device corresponds to multiple exposure parameters and multiple resolution parameters). Specifically, the single image capturing device generates the multiple instant images based on the multiple first exposure parameters and the multiple first resolution parameters.

[0082] In some embodiments, multiple image capturing devices can capture instant images by setting different exposure parameters and resolution parameters. Specifically, the multiple image capturing devices generate the multiple instant images based on the second exposure parameters and the second resolution parameters respectively corresponding to the multiple image capturing devices.

[0083] From the above description, it can be seen that the synthetic image generating device 1 provided by the present disclosure performs corresponding photometric operations by analyzing the eye fixation position of the user, and determines the components of the synthetic image based on the brightness value of the eye fixation position. The synthetic image generating device 1 provided by the present disclosure provides that important parts in the synthetic image are composed of instant images with higher resolutions, and less important parts in the synthetic image are composed of instant images with lower resolutions, improving the efficiency of the synthetic image. Since the synthetic image generating device 1 provided by the present disclosure solves the problem that the prior art cannot be applied to instant display, it improves the service experience of the user.

[0084] The second embodiment of the present invention is a synthetic image generating method, and its flowchart is shown in Figure 5 ... The synthetic image generating method 500 is applicable to an electronic device, for example: the synthetic image generating device 1 described in the first embodiment. The synthetic image generating method 500 generates a synthetic image through steps S501 to S507.

[0085] In step S501, the electronic device determines an eye fixation position corresponding to a user.

[0086] Next, in step S503, the electronic device determines a region of interest corresponding to a plurality of instant images based on the eye gaze position, where each of the plurality of instant images corresponds to an exposure value and a resolution.

[0087] Next, in step S505, the electronic device generates a composite image based on the region of interest and a non - region of interest corresponding to the plurality of instant images, where the region of interest and the non - region of interest of the composite image are generated based on the plurality of instant images corresponding to different resolutions.

[0088] Finally, in step S507, the electronic device transmits the composite image to a display device for an instant display operation.

[0089] In some embodiments, the step of generating the composite image further includes the following steps: generating a plurality of first regional pixel values corresponding to the region of interest based on the plurality of instant images corresponding to a first resolution; generating a plurality of second regional pixel values corresponding to the non - region of interest based on the plurality of instant images corresponding to a second resolution; and synthesizing the plurality of first regional pixel values and the plurality of second regional pixel values to generate the composite image, where the first resolution is higher than the second resolution.

[0090] In some embodiments, the plurality of instant images corresponding to the second resolution are generated by the following steps: calculating a luminance value of a photometric image at the eye gaze position; selecting at least one second instant image from the plurality of instant images based on the luminance value of the photometric image at the eye gaze position and the luminance values of the plurality of instant images corresponding to the region of interest; and performing a down - resolution operation on the at least one second instant image to generate the plurality of instant images corresponding to the second resolution.

[0091] In some embodiments, the step of determining the eye gaze position of the user further includes the following steps: determining the eye gaze position of the user based on eye tracking information of the user in a photometric image.

[0092] In some embodiments, the plurality of instant images are generated by the following steps: calculating a luminance value of the photometric image at the eye gaze position; determining the exposure value and the resolution corresponding to each of the plurality of instant images based on the luminance value of the photometric image at the eye gaze position, where the plurality of instant images are generated by at least one image capture device based on the exposure value and the resolution corresponding to each of the plurality of instant images.

[0093] In some embodiments, the step of determining the region of interest further includes the following steps: calculating a brightness value of the photometric image at the eye fixation position; generating a plurality of target pixel positions corresponding to the brightness value in the photometric image; and determining the region of interest based on the plurality of target pixel positions.

[0094] In some embodiments, the step of determining the region of interest further includes the following steps: identifying a target object corresponding to the eye fixation position in the photometric image; generating a plurality of target pixel positions corresponding to the target object in the photometric image; and determining the region of interest based on the plurality of target pixel positions.

[0095] In some embodiments, the plurality of instantaneous images are generated by a single image capturing device, and the single image capturing device corresponds to a plurality of first exposure parameters and a plurality of first resolution parameters, and the plurality of instantaneous images are generated by the following steps: generating the plurality of instantaneous images by the single image capturing device based on the plurality of first exposure parameters and the plurality of first resolution parameters.

[0096] In some embodiments, the plurality of instantaneous images are generated by a plurality of image capturing devices, and each of the plurality of image capturing devices corresponds to a second exposure parameter and a second resolution parameter, and the plurality of instantaneous images are generated by the following steps: generating the plurality of instantaneous images by the plurality of image capturing devices based on the second exposure parameters and the second resolution parameters corresponding to each of the plurality of image capturing devices.

[0097] In addition to the above steps, the second embodiment can also perform all the operations and steps of the synthetic image generating device 1 described in the first embodiment, having the same functions and achieving the same technical effects. Those of ordinary skill in the technical field to which the present invention pertains can directly understand how the second embodiment performs these operations and steps based on the above first embodiment, having the same functions and achieving the same technical effects, so they will not be elaborated.

[0098] The synthetic image generation method described in the second embodiment can be implemented by a computer program having a plurality of instructions. Each computer program can be a file that can be transmitted over a network or can be stored in a non-transitory computer-readable storage medium. For each computer program, after the plurality of instructions included therein are loaded into an electronic device (e.g., the synthetic image generation device 1), the computer program executes the synthetic image generation method described in the second embodiment. The non-transitory computer-readable storage medium can be an electronic product, such as a read only memory (ROM), a flash memory, a floppy disk, a hard disk, a compact disk (CD), a USB drive, a database accessible via a network, or any other storage medium known to those of ordinary skill in the art to which the present invention pertains and having the same function.

[0099] It should be noted that in the description and claims of the present invention patent, some terms (including: resolution, regional pixel value, exposure parameter, resolution parameter, etc.) are prefixed with "first" or "second". The plurality of "first" or "second" are only used to distinguish different terms. For example, the "first" and "second" in the first resolution and the second resolution are only used to represent the resolutions used in different operations.

[0100] In summary, the synthetic image generation technology provided by the present disclosure (at least including the device, method, and its non-transitory computer-readable storage medium) performs corresponding photometry operations by analyzing the user's eye fixation position, and determines the components of the synthetic image based on the luminance value of the eye fixation position. The synthetic image generation technology provided by the present disclosure provides for composing the important parts in the synthetic image with high-resolution real-time images and composing the less important parts in the synthetic image with low-resolution real-time images, improving the efficiency of the synthetic image. Since the synthetic image generation technology provided by the present disclosure solves the problem that the prior art cannot be applied to real-time display, it improves the user's service experience.

[0101] The above embodiments are only used to illustrate some implementation modes of the present invention and to explain the technical features of the present invention, rather than to limit the protection scope and range of the present invention. Any changes or equivalent arrangements that can be easily made by those of ordinary skill in the art to which the present invention pertains belong to the scope claimed by the present invention, and the scope of the protection of the present invention is subject to the claims.

Claims

1. A synthetic image generation device, characterized in that, Comprising: A transceiver interface; and A processor, electrically connected to the transceiver interface, and configured to perform the following operations: Determine a gaze position of an eye corresponding to a user; Based on the gaze position of the eye, determine a region of interest corresponding to a plurality of live images, wherein Each of the plurality of live images corresponds to an exposure value and a resolution; Based on the region of interest and a non-interest region corresponding to the plurality of live images, generate a composite image, wherein the region of interest and the non-interest region of the composite image are generated based on the plurality of live images corresponding to different resolutions; and Transmit the composite image to a display device for a live display operation.

2. The synthetic image generating device according to claim 1, wherein Wherein generating the composite image further comprises the following operations: Based on the plurality of live images corresponding to a first resolution, generate a plurality of first region pixel values corresponding to the region of interest; Based on the plurality of live images corresponding to a second resolution, generate a plurality of second region pixel values corresponding to the non-interest region; And Composite the plurality of first region pixel values and the plurality of second region pixel values to generate the composite image, wherein the first resolution is higher than the second resolution.

3. The synthetic image generating device according to claim 2, wherein Wherein the plurality of live images corresponding to the second resolution are generated by the following operations: Calculate a luminance value of a photometric image at the gaze position of the eye; Based on the luminance value of the photometric image at the gaze position of the eye and the luminance values of the plurality of live images corresponding to the region of interest, select at least one second live image from the plurality of live images; And Perform a downsampling operation on the at least one second live image to generate the plurality of live images corresponding to the second resolution.

4. The synthetic image generating device according to claim 1, wherein, Wherein determining the gaze position of the user further comprises the following operations: Based on eye tracking information of the user in a photometric image, determine the gaze position of the user.

5. The synthetic image generating device according to claim 4, wherein, Wherein the plurality of live images are generated by the following operations: Calculate a luminance value of the photometric image at the gaze position of the eye; and Based on the luminance value of the photometric image at the gaze position of the eye, determine the exposure value and the resolution corresponding to each of the plurality of live images, wherein the plurality of live images are generated by at least one image capture device based on the exposure value and the resolution corresponding to each of the plurality of live images.

6. The synthetic image generating device according to claim 4, wherein, Wherein determining the region of interest further comprises the following operations: Calculate a luminance value of the photometric image at the gaze position of the eye; Generate a plurality of target pixel positions in the photometric image corresponding to the luminance value; And Based on the plurality of target pixel positions, determine the region of interest.

7. The synthetic image generating device according to claim 4, wherein Wherein determining the region of interest further comprises the following operations: Identify a target object corresponding to the gaze position in the photometric image; Generate a plurality of target pixel positions in the photometric image corresponding to the target object; And Based on the plurality of target pixel positions, determine the region of interest.

8. The synthetic image generating device according to claim 1, wherein, The multiple instant images are generated by a single image capturing device, the single image capturing device corresponding to a plurality of first exposure parameters and a plurality of first resolution parameters, and the multiple instant images are generated by the following operations: The single image capturing device generates the multiple instant images based on the plurality of first exposure parameters and the plurality of first resolution parameters.

9. The synthetic image generating device according to claim 1, characterized in that, The multiple instant images are generated by a plurality of image capturing devices, the plurality of image capturing devices each corresponding to a second exposure parameter and a second resolution parameter, and the multiple instant images are generated by the following operations: The plurality of image capturing devices generate the multiple instant images based on the second exposure parameter and the second resolution parameter corresponding to each of the plurality of image capturing devices.

10. The synthetic image generating device according to claim 1, wherein The display device is a head-mounted display device, and the head-mounted display device is worn by the user.

11. A method for generating a synthetic image, characterized in that, For an electronic device, the method for generating a composite image includes the following steps: Determine a gaze position of one eye corresponding to a user; Based on the gaze position, determine a region of interest corresponding to the multiple instant images, wherein each of the multiple instant images corresponds to an exposure value and a resolution; Generate a composite image based on the region of interest and a non-region of interest corresponding to the multiple instant images, wherein the region of interest and the non-region of interest of the composite image are generated based on the multiple instant images corresponding to different resolutions; and Transmit the composite image to a display device for an instant display operation.

12. The synthetic image generation method according to claim 11, wherein, Wherein generating the composite image further includes the following steps: Generate a plurality of first region pixel values corresponding to the region of interest based on the multiple instant images corresponding to a first resolution; Generate a plurality of second region pixel values corresponding to the non-region of interest based on the multiple instant images corresponding to a second resolution; And Combine the plurality of first region pixel values and the plurality of second region pixel values to generate the composite image, wherein the first resolution is higher than the second resolution.

13. The synthetic image generation method according to claim 12, wherein Wherein the multiple instant images corresponding to the second resolution are generated by the following steps: Calculate a luminance value of a photometric image at the gaze position; Based on the luminance value of the photometric image at the gaze position and the luminance values of the multiple instant images corresponding to the region of interest, select at least one second instant image from the multiple instant images; And Perform a downsampling operation on the at least one second instant image to generate the multiple instant images corresponding to the second resolution.

14. The synthetic image generation method according to claim 11, characterized in that, Wherein determining the gaze position of the user further includes the following steps: Determine the gaze position of the user based on eye tracking information of the user in a photometric image.

15. The synthetic image generation method according to claim 14, wherein Wherein the multiple instant images are generated by the following steps: Calculate a luminance value of the photometric image at the gaze position; and Based on the luminance value of the photometric image at the eye fixation position, determine the exposure value and the resolution corresponding to each of the plurality of instant images, wherein the plurality of instant images are generated by at least one image capturing device based on the exposure value and the resolution corresponding to each of the plurality of instant images.

16. The synthetic image generation method according to claim 14, wherein, Wherein determining the region of interest further comprises the following steps: Calculate a luminance value of the photometric image at the eye fixation position; Generate a plurality of target pixel positions in the photometric image corresponding to the luminance value; And Based on the plurality of target pixel positions, determine the region of interest.

17. The synthetic image generation method according to claim 14, wherein Wherein determining the region of interest further comprises the following steps: Identify a target object corresponding to the eye fixation position in the photometric image; Generate a plurality of target pixel positions in the photometric image corresponding to the target object; And Based on the plurality of target pixel positions, determine the region of interest.

18. The synthetic image generation method according to claim 11, wherein, Wherein the plurality of instant images are generated by a single image capturing device, the single image capturing device corresponding to a plurality of first exposure parameters and a plurality of first resolution parameters, and the plurality of instant images are generated by the following steps: The single image capturing device generates the plurality of instant images based on the plurality of first exposure parameters and the plurality of first resolution parameters.

19. The synthetic image generation method according to claim 11, wherein Wherein the plurality of instant images are generated by a plurality of image capturing devices, the plurality of image capturing devices each corresponding to a second exposure parameter and a second resolution parameter, and the plurality of instant images are generated by the following steps: The plurality of image capturing devices generate the plurality of instant images based on the second exposure parameter and the second resolution parameter corresponding to each of the plurality of image capturing devices.

20. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores a computer program, the computer program comprising a plurality of program instructions, the computer program performing a synthetic image generation method after being loaded into an electronic device, the synthetic image generation method comprising the following steps: Determine an eye fixation position corresponding to a user; Based on the eye fixation position, determine a region of interest corresponding to a plurality of instant images, wherein each of the plurality of instant images corresponds to an exposure value and a resolution; Generate a synthetic image based on the region of interest and a non-region of interest corresponding to the plurality of instant images, wherein the region of interest and the non-region of interest of the synthetic image are generated based on the plurality of instant images corresponding to different resolutions; and Transmit the synthetic image to a display device for an instant display operation.

Images