System and method for converting two-dimensional images into multi-view images according to user level

Abstract

A system and method for converting two-dimensional images into multi-view images based on user level. The system uses a cloud server to generate conversion parameters based on the client's user level. The conversion server then performs image enhancement calculations on the initial depth map of the target image transmitted by the client based on these parameters to generate a final depth image. It also determines the characteristics of the grating film based on the characteristic parameters of the stereoscopic display device and generates multi-view images that conform to the grating film characteristics based on the final depth image. This allows the stereoscopic display device to display various images from different perspectives using optical principles after the grating-stitched multi-view images are superimposed, thus presenting a stereoscopic image. This method achieves the technical effect of providing conversion services with different payment options to generate multi-view images with varying comfort levels and resolutions.

Description

Technical Field

[0001] An image conversion system and method thereof, particularly a system and method for converting two-dimensional images into multi-view images according to user level. Background Technology

[0002] In recent years, image processing technology has continued to advance, representing humanity's ongoing pursuit of a more realistic visual experience. Beyond advancements in general image and color processing techniques, the processing of multi-view stereoscopic images has further enhanced the visual experience of three-dimensional space. For example, stereoscopic image display technology is widely used in the entertainment industry, medical systems, and many other fields.

[0003] However, comfortable stereoscopic images require multi-view images, but the high cost of multi-view cameras makes comfortable stereoscopic displays difficult to achieve. To address this issue, depth-based image generation techniques are used. These techniques utilize optical projection principles to preprocess 2D images with depth maps, perform 3D warping calculations, and hole filling to generate 2D images with other perspectives. These original 2D images are then combined with other 2D images to create multi-view images suitable for stereoscopic display, thus solving this problem.

[0004] However, because current depth image generation techniques often produce distorted 2D images from other perspectives, the quality of stereoscopic images displayed using multi-view images generated by these techniques is insufficient to meet the needs of comfortable viewing. Therefore, solutions to improve depth image generation techniques have been proposed, but most of these solutions require sufficient computing power.

[0005] As internet services become more widespread, manufacturers often offer free versions to increase brand awareness. Consequently, more and more people become accustomed to using free services, and few are willing to pay for them. For internet services that require significant investment, such as image conversion services that improve depth image generation technology, manufacturers cannot recoup their development costs in a short period.

[0006] In conclusion, it is clear that existing technologies have long suffered from the problem that image conversion services, which require financial investment, are not suitable for free provision. Therefore, it is necessary to propose improved technical solutions to address this issue. Summary of the Invention

[0007] In view of the problem that existing image conversion services are not suitable for free provision, this invention discloses a system and method for converting two-dimensional images into multi-view images according to user level, wherein:

[0008] The system disclosed in this invention for converting two-dimensional images into multi-view images based on user level includes at least: a client for selecting a target image and setting device information, wherein the target image is a two-dimensional image; a cloud server for receiving the target image and device information, obtaining the client's user level, generating conversion parameters based on the user level, and converting the device information into characteristic parameters of a stereoscopic display device, wherein the characteristic parameters include the number of disparity images; and a conversion server for receiving the target image, characteristic parameters, and conversion parameters, generating an initial depth map of the target image using a deep learning model, and selectively performing at least [specific actions] on the initial depth map based on the conversion parameters. Image enhancement calculations are performed to generate a final depth image. Based on the characteristic parameters of the stereoscopic display device, the characteristics of the grating film of the stereoscopic display device are determined, and multi-view images that conform to the characteristics of the grating film of the stereoscopic display device are generated based on the final depth image. The multi-view images include display images from multiple different perspectives. The stereoscopic display device, corresponding to the device information, includes a grating film. The grating film contains a grating and has grating film characteristics. The stereoscopic display device is used to download the multi-view images after the conversion server generates them, and to display the multi-view images. The multi-view images are superimposed using gratings, thereby presenting each display image corresponding to each perspective in the multi-view images from different perspectives through optical principles.

[0009] The method for converting a two-dimensional image into a multi-view image based on user level disclosed in this invention includes at least the following steps: a client selects a target image and sets device information for a stereoscopic display device, and transmits the target image and device information to a cloud server, wherein the target image is a two-dimensional image; the cloud server obtains the user level of the client and generates conversion parameters based on the user level; the cloud server converts the device information into characteristic parameters of the stereoscopic display device, and transmits the target image, characteristic parameters, and conversion parameters to a conversion server, wherein the characteristic parameters include the number of disparity images; the conversion server uses a deep learning model to generate an initial depth map of the target image; the conversion server selects to selectively perform at least image enhancement calculations on the initial depth map according to the conversion parameters to generate a final depth image; the conversion server determines the grating film characteristics of the stereoscopic display device according to the characteristic parameters; the conversion server generates a multi-view image that matches the grating film characteristics of the stereoscopic display device based on the final depth image, wherein the multi-view image includes multiple display images from different perspectives; the stereoscopic display device downloads the multi-view image after the conversion server generates it; the stereoscopic display device displays the multi-view image and uses the grating of the display of the stereoscopic display device to superimpose the multi-view image, thereby presenting each display image corresponding to each perspective in the multi-view image from each different perspective through optical principles.

[0010] The system and method disclosed in this invention are as described above. The difference between this invention and the prior art lies in that, after the cloud server generates characteristic parameters based on the device information transmitted by the client and conversion parameters based on the user level of the client, the conversion server selects image enhancement calculations that can be performed on the initial depth map of the target image transmitted by the client based on the conversion parameters to generate a final depth image. Based on the characteristic parameters of the stereoscopic display device, the characteristics of the grating film of the stereoscopic display device are determined, and multi-view images that conform to the characteristics of the grating film of the stereoscopic display device are generated based on the final depth image. The stereoscopic display device downloads and displays the multi-view images, so that after the grating superimposes the multi-view images, the stereoscopic display device can display each display image in the multi-view images from different angles through optical principles to present stereoscopic images. This solves the problems existing in the prior art and can achieve the technical effect of providing multi-view images with different comfort levels and resolutions of stereoscopic images with different payment plans. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the components of the computing device proposed in this invention.

[0012] Figure 2 This is a schematic diagram of the components of the system proposed in this invention that converts two-dimensional images into multi-view images according to user level.

[0013] Figure 3A This is a flowchart of the method for converting two-dimensional images into multi-view images according to user level, as proposed in this invention.

[0014] Figure 3B This is a flowchart of the method for generating the final depth image proposed in this invention.

[0015] Figure 3C This is a flowchart of the method for downloading multi-view images using the stereoscopic display device proposed in this invention.

[0016] Figure 3D The flowchart illustrates the method for setting client user levels on a cloud server as proposed in this invention.

[0017] Figure 4 This is a schematic diagram of the multi-view images arranged in a sub-pixel-level staggered manner according to an embodiment of the present invention.

[0018] The annotations in the attached figures are explained as follows:

[0019] 100: Computing devices

[0020] 110: Memory

[0021] 120: Input Unit

[0022] 130: Communication Interface

[0023] 140: Storage medium

[0024] 150: Output Unit

[0025] 170: Processor

[0026] 190: Bus

[0027] 210: Client

[0028] 220: Cloud server

[0029] 221: Cloud Transmission Module

[0030] 223: Payment Processing Module

[0031] 225: Hierarchy Setting Module

[0032] 227: Parameter Generation Module

[0033] 229: Image Storage and Retrieval Module

[0034] 230: Conversion Server

[0035] 231: Service Transport Module

[0036] 233: Depth Generation Module

[0037] 235: Depth Enhancement Module

[0038] 237: Image Generation Module

[0039] 250: Stereoscopic display device

[0040] 251: Device Transmission Module

[0041] 253: Device display module

[0042] 257: Gate Width Calculation Module

[0043] 259: Gate Width Adjustment Module

[0044] Step 301: The client selects a payment plan and sends the payment plan to the cloud server.

[0045] Step 303: The cloud server provides the client with the ability to complete the payment task corresponding to the payment plan.

[0046] Step 305: The cloud server sets the client's user tier according to the payment plan.

[0047] Step 310: The client selects the target image and sets the device information, and then transmits the target image and device information to the cloud server.

[0048] Step 321: The cloud server obtains the client's user level and generates conversion parameters based on the user level.

[0049] Step 325: The cloud server converts the device information into characteristic parameters and transmits the target image, characteristic parameters, and conversion parameters to the conversion server.

[0050] Step 340: The conversion server generates the initial depth map of the target image.

[0051] Step 350: The conversion server selectively performs image enhancement calculations on the initial depth map based on the conversion parameters to generate the final depth image.

[0052] Step 351: The transformation server performs a linear transformation on the brightness range of the depth map to generate a first intermediate image.

[0053] Step 353: The conversion server performs low-pass filtering on the first intermediate image to generate the second intermediate image.

[0054] Step 355: The transformation server performs a visual depth-constrained linear transformation on the second intermediate image to produce the final depth image.

[0055] Step 360: The conversion server determines the characteristics of the raster film in the stereoscopic display device based on the characteristic parameters.

[0056] Step 370: The conversion server generates a multi-view image that conforms to the characteristics of the raster film of the stereoscopic display device based on the final depth image. The multi-view image contains display images from multiple different perspectives.

[0057] Step 380: Download multi-view images to the stereoscopic display device.

[0058] Step 381: The conversion server generates conversion completion information and transmits the conversion completion information and multi-view images to the cloud server.

[0059] Step 383: The cloud server provides the 3D display device with a query to see if the conversion completion information exists.

[0060] Step 385: Download multi-view images to the stereoscopic display device

[0061] Step 390: The stereoscopic display device displays multi-view images and uses gratings to overlay the multi-view images, thereby displaying corresponding display images in the multi-view images from different angles through optical principles. Detailed Implementation

[0062] The features and implementation methods of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. The content is sufficient to enable any person skilled in the art to easily and fully understand the technical means used by the present invention to solve the technical problem and to implement it accordingly, thereby achieving the effects that the present invention can achieve.

[0063] This invention allows a client to upload a two-dimensional (planar) target image to a cloud server. The conversion server then converts the target image into a multi-view image with a level of detail corresponding to the client's user level. The multi-view image is then superimposed using a grating on a stereoscopic display device to present the target image in three dimensions.

[0064] The client, cloud server, conversion server, and stereoscopic display device implementing this invention can be a computing device. The computing device of this invention includes, but is not limited to, one or more processing modules, one or more memory modules, and buses connecting different hardware components (including memory modules and processing modules). Through the included hardware components, the computing device can load and execute an operating system, allowing the operating system to run on the computing device, and can also execute software or programs. The computing device also includes a housing, within which the aforementioned hardware components are disposed.

[0065] The bus of the computing device proposed in this invention can include one or more types, such as a data bus, address bus, control bus, expansion bus, and / or local bus. The bus of the computing device includes, but is not limited to, Industry Standard Architecture (ISA) bus, Peripheral Component Interconnect (PCI) bus, Video Electronics Standards Association (VESA) local bus, and serial Universal Serial Bus (USB), PCI Express (PCI-E / PCIe) bus, etc.

[0066] The processing module of the computing device proposed in this invention is coupled to a bus. The processing module includes a register set or register space, which may be entirely located on the processing chip of the processing module, or wholly or partially located outside the processing chip and coupled to the processing chip via dedicated electrical connections and / or via a bus. The processing module may be a central processing unit, a microprocessor, or any suitable processing element. If the computing device is a multiprocessor device, that is, the computing device contains multiple processing modules, then the processing modules contained in the computing device are identical or similar and are coupled and communicate via a bus. In some embodiments, the processing module may interpret a computer instruction or a series of multiple computer instructions to perform specific operations or calculations, such as mathematical operations, logical operations, data comparison, copying / moving data, etc., thereby driving other hardware components in the computing device or running an operating system or executing various programs and / or modules. Computer instructions can be assembly language instructions, instruction set architecture instructions, machine instructions, machine-dependent instructions, microinstructions, firmware instructions, or source code or object code written in any combination of one or more programming languages. Computer instructions can be executed entirely on a single computing device, partially on a single computing device, or partially on one computing device and partially on another connected computing device. The aforementioned programming languages ​​include object-oriented programming languages ​​such as Common Lisp, Python, C++, Objective-C, Smalltalk, Delphi, Java, Swift, C#, Perl, Ruby, etc., as well as conventional procedural programming languages ​​such as C or other similar programming languages.

[0067] Computing devices typically include one or more chipsets. The processing modules of the computing device can be coupled to or electrically connected to the chipset via a bus. A chipset consists of one or more integrated circuits (ICs), including a memory controller and peripheral input / output (I / O) controllers, etc. That is, the memory controller and I / O controllers can be contained within a single IC or implemented using two or more ICs. Chipsets typically provide I / O and memory management functions, as well as multiple general-purpose and / or special-purpose registers, timers, etc., which can be accessed or used by one or more processing modules coupled to or electrically connected to the chipset. In some embodiments, the chipset may also be part of the processing module.

[0068] The processing module of a computing device can also access data in the memory modules and mass storage areas installed on the computing device through the memory controller. The aforementioned memory modules include any type of volatile memory and / or non-volatile memory (NVRAM), such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Read-Only Memory (ROM), Flash memory, etc. The aforementioned mass storage areas can contain any type of storage device or storage medium, such as hard disks, optical discs, flash drives, memory cards, solid-state drives (SSDs), or any other storage device. In other words, the memory controller can access data in static random access memory, dynamic random access memory, flash memory, hard disks, and solid-state drives.

[0069] The processing module of a computing device can also connect and communicate with peripheral devices or interfaces such as peripheral output devices, peripheral input devices, communication interfaces, and various data or signal receiving devices via a peripheral input / output controller and a peripheral input / output bus. Peripheral input devices can be any type of input device, such as a keyboard, mouse, trackball, touchpad, joystick, etc. Peripheral output devices can be any type of output device, such as a monitor, printer, etc. Peripheral input devices and peripheral output devices can also be the same device, such as a touchscreen. Communication interfaces can include wireless communication interfaces and / or wired communication interfaces. Wireless communication interfaces can include interfaces supporting wireless local area networks (such as Wi-Fi, Zigbee, etc.), Bluetooth, infrared, near-field communication (NFC), 3G / 4G / 5G and other mobile communication networks (mobile networks) or other wireless data transmission protocols. Wired communication interfaces can be Ethernet devices, DSL modems, cable modems, asynchronous transfer mode (ATM) devices, or fiber optic communication interfaces and / or components, etc. The data or signal receiving device may include a GPS receiver or a physiological signal receiver, wherein the physiological signals received by the physiological signal receiver include, but are not limited to, heartbeat, blood oxygenation, etc. The processing module may periodically poll various peripheral devices and interfaces, enabling the computing device to input and output data through various peripheral devices and interfaces, and also to communicate with another computing device having the hardware components described above.

[0070] The following is a preliminary step. Figure 1 The schematic diagram of the computing device proposed in this invention is used to illustrate the apparatus for implementing this invention. For example... Figure 1 As shown, the computing device 100 of the present invention includes a memory 110, an input unit 120, a communication interface 130, a storage medium 140, an output unit 150, a processor 170, and a bus 190. The processor 170 is interconnected with the memory 110, the input unit 120, the communication interface 130, the storage medium 140, and the output unit 150 via the bus 190.

[0071] The memory 110 can store one or more sets of computer instructions.

[0072] The communication interface 130 can be connected to external network storage devices or servers, and request and download data from the connected network devices.

[0073] Input unit 120 can provide input data through peripheral input devices of computing device 100. For example, input unit 120 can input data through a keyboard, mouse, touchpad, or touch screen connected to or provided on computing device 100.

[0074] Storage medium 140 can store data or signals downloaded from communication interface 130, data or signals provided to processor 170 or required for processor 170 to operate, and data or signals generated by processor 170.

[0075] The output unit 150 can also output the data generated by the processor 170 through the peripheral output device of the computing device 100. For example, the output unit 150 can display the data generated by the processor 170 through a display or touch screen connected to or provided on the computing device 100.

[0076] The processor 170 can execute computer instructions stored in the memory 110 and can generate modules after executing the computer instructions; in another embodiment, each module in the processor 170 can be generated by one or more circuits and / or complete or partial chips and other hardware elements, that is, the processor 170 includes hardware elements that make up each module. In other words, each module included in the processor 170 can be a software module or a hardware module, and the present invention has no particular limitation.

[0077] Continue with Figure 2 The present invention is illustrated by schematic diagrams of components of a system for converting two-dimensional images into multi-view images based on user hierarchy. For example... Figure 2As shown, the system of the present invention includes a client 210, a cloud server 220, a conversion server 230, and a stereoscopic display device 250.

[0078] Client 210 is responsible for selecting the target image. The target image selected by client 210 is a two-dimensional image, that is, a planar image. It should be noted that the target image selected by client 210 is not limited to a single image, but can also be multiple images.

[0079] The client 210 is also responsible for setting device information. The device information set by the client 210 corresponds to the stereoscopic display device 250 and can represent the stereoscopic display device 250. For example, the device information may include device identification data and model number of the stereoscopic display device 250. The device identification data may be data such as the serial number of the stereoscopic display device 250, but the present invention is not limited thereto.

[0080] The client 210 is also responsible for transmitting the selected target image and the set device information to the cloud server 220. Generally, when transmitting the target image and device information, the client 210 may also transmit the user identification data of the client 210 to the cloud server 220 at the same time, but the present invention is not limited thereto.

[0081] Client 210 can also select a payment plan and transmit the selected payment plan to cloud server 220, and can complete the payment operation corresponding to the selected payment plan through cloud server 220. The payment plan selected by client 210 includes a payment mode and a payment amount. For example, the payment mode can be a one-time payment or a monthly payment, and the payment amount can be a basic fee or a basic fee plus value-added fees. However, the payment plan, payment mode, and payment amount proposed in this invention are not limited to the above.

[0082] The cloud server 220 is responsible for providing the target image to the conversion server 230 and can store the multi-view images generated by the conversion server 230. In some embodiments, the cloud server 220 may include a cloud transmission module 221, a layer setting module 225, and a parameter generation module 227, and may also include an attachable paid processing module 223 and an image storage and retrieval module 229.

[0083] The cloud transmission module 221 can receive target images and device information transmitted by the client 210, as well as user identification data transmitted by the client 210, and can transmit the target images to the conversion server 230. In some embodiments, the cloud transmission module 221 can also receive multi-view images transmitted by the conversion server 230, and can receive image query requests transmitted by the stereoscopic display device 250, and can transmit multi-view images to the stereoscopic display device 250.

[0084] The payment processing module 223 can enable the client 210 to complete the payment operation corresponding to the payment scheme received by the cloud transmission module 221.

[0085] The tier setting module 225 can set the user tier of client 210 to the tier corresponding to the payment scheme received by cloud transmission module 221 after the payment processing module 223 provides the client 210 with the payment operation completed. It can also store the user identification data received by cloud transmission module 221 and the set user tier as a single data entry. For example, the user tier can include three levels: non-member, basic member, and advanced member. The tier setting module 225 can set the user tier to non-member before the user of client 210 pays the fee, and set the user tier to basic member or advanced member respectively when the user of client 210 pays the basic fee or the basic fee plus the value-added fee. However, this invention is not limited to these limitations.

[0086] The parameter generation module 227 is responsible for determining the characteristic parameters corresponding to the stereoscopic display device 250 based on the device information received by the cloud transmission module 221. The characteristic parameters generated by the parameter generation module 227 may include the resolution, diagonal size, width and height, and pixel pitch of the stereoscopic display device 250's display. For example, when the device information includes the model number of the stereoscopic display device 250, the parameter generation module 227 can read the corresponding characteristic parameters from a pre-stored model data table based on the model number included in the device information. The model data table can be stored in the storage medium of the cloud server 220 or in a network device connected to the cloud server 220; this invention has no particular limitation.

[0087] The parameter generation module 227 is also responsible for reading the corresponding user level based on the user identification data received by the cloud transmission module 221, and can generate conversion parameters based on the read user level. The conversion parameters generated by the parameter generation module 227 can represent the image enhancement calculations that the conversion server 230 needs to perform.

[0088] When the image storage module 229 receives the multi-view image transmitted by the conversion server 230 from the cloud transmission module 221, it stores the multi-view image received by the cloud transmission module 221 and the device identification data of the stereoscopic display device 250 as a single data entry in the device image correspondence table. The device identification data stored by the image storage module 229 can be included in the device information received by the cloud transmission module 221, or it can be included in the conversion completion information corresponding to the multi-view image transmitted by the conversion server 230 and received by the cloud transmission module 221.

[0089] When the image storage and retrieval module 229 receives an image query request from the stereoscopic display device 250 via the cloud transmission module 221, it can read the corresponding multi-view image from the stored device image correspondence table based on the device identification data contained in the image query request, and transmit the read multi-view image to the stereoscopic display device 250 via the cloud transmission module 221.

[0090] The conversion server 230 is responsible for generating multi-view images that match the raster characteristics of the stereoscopic display device 250 based on the target image selected by the cloud server 220. In some embodiments, the processor of the conversion server 230 may include modules such as a service transmission module 231, a depth generation module 233, a depth enhancement module 235, and an image generation module 237, and can generate multi-view images through the above modules.

[0091] The service transmission module 231 is responsible for receiving the target image, characteristic parameters, and conversion parameters transmitted by the cloud server 220 through the communication interface of the conversion server 230. The service transmission module 231 can also transmit the conversion completion information and multi-view images generated by the image generation module 237 to the cloud server 220 through the communication interface of the conversion server 230.

[0092] The depth generation module 233 is responsible for generating a depth map of the target image received by the service transmission module 231 using a deep learning model. In this invention, the depth map generated by the depth generation module 233 is also referred to as the "initial depth map". The depth map generated by the depth generation module 233 is a grayscale image, and the value of each pixel in the depth map represents the depth of that point in the scene, that is, the distance from the camera.

[0093] It should be noted that since the target image can be a single image or multiple images, the deep learning model used by the depth generation module 233 may also be different. For example, when the target image is a single image, the deep learning model can be a combination of Transformer and Convolutional Neural Network (CNN), such as DPT (Dense Prediction Transformer) or MiDaS (Mixed-Dataset Depth Estimation); it can also be a self-supervised learning model, such as Monodepth2; or it can be a NeRF (Neural Radiance Fields) model, such as NeRF, Mip-NeRF, or Instant-NGP. When the target image consists of multiple images, the deep learning model calculates the depth map through parallax, feature matching, and geometric reconstruction. For example, the deep learning model can be a deep learning method based on CNN and Transformer for multi-view depth estimation, such as MVSNet (Multi-View Stereo Network), DeepMVS, and MVDepthNet (Multi-View Depth Estimation Network); it can also be a combination of multi-view stereo parallax (MVS) and deep learning; or it can be the NeRF model mentioned above.

[0094] The depth enhancement module 235 is responsible for selecting, based on the conversion parameters received by the service transmission module 231, to perform one or more image enhancement calculations on the initial depth map generated by the depth generation module 233 to generate another depth map. In this invention, the depth map generated by the depth enhancement module 235 is also referred to as the "final depth image." Alternatively, the depth enhancement module 235 may choose not to perform any image enhancement calculations and directly use the initial depth map as the final depth image based on the conversion parameters. Through image enhancement calculations, the initial depth map can be depth-corrected and enhanced, and the visual stereoscopic effect can be adjusted to best match the visual perception.

[0095] For example, when the transformation parameters represent image enhancement calculations that perform brightness expansion and equalization on the depth map, the depth enhancement module 235 can choose to perform linear transformations on each brightness range of the depth map to generate another depth map that can obtain more stereoscopic depth details and increase image color contrast; when the transformation parameters represent image enhancement calculations that perform low-pass filtering on the depth map, the depth enhancement module 235 can also generate another depth map that has been smoothed; and when the transformation parameters represent image enhancement calculations that perform depth-constrained linear transformations on the depth map, the depth enhancement module 235 can also generate another depth map that conforms to the rules of visual perception. It should be noted that the depth map used in the various image enhancement calculations performed by the depth enhancement module 235 can be an initial depth map or a depth map generated after image enhancement calculations. In other words, the depth enhancement module 235 can perform different image enhancement calculations on the depth map generated by the image enhancement calculations again, and so on, to generate the final depth image. In practical terms, the depth enhancement module 235 can perform one, two, or all of the above-mentioned image enhancement calculations such as brightness expansion and equalization, low-pass filtering, and visual depth constraint linear transformation to generate the final depth image. Moreover, there is no restriction on the order in which the above-mentioned image enhancement calculations are performed. For example, the depth enhancement module 237 can perform only two image enhancement calculations. For example, it can first perform a brightness expansion and equalization image enhancement calculation on the depth map to generate an intermediate image, and then perform a low-pass filtering image enhancement calculation on the intermediate image to generate the final depth image. Or, it can first perform a brightness expansion and equalization or low-pass filtering image enhancement calculation on the depth map to generate an intermediate image, and then perform a visual depth constraint linear transformation image enhancement calculation on the intermediate image to generate the final depth image.

[0096] The image generation module 237 is responsible for determining the raster film characteristics of the stereoscopic display device 250 based on the characteristic parameters received by the service transmission module 231. The raster film characteristics mentioned in this invention may include the number of parallax images, parallax range, and viewing distance of the display of the stereoscopic display device 250. The number of parallax images is the number of views that the stereoscopic display device 250 can provide, such as 2, 4, 8, 16, etc. For example, the image generation module 237 can first calculate the number of parallax images of the stereoscopic display device 250 based on the width of the display and the width of a single raster unit in the raster film of the stereoscopic display device 250 in the characteristic parameters to determine the raster film characteristics of the stereoscopic display device 250. For example, the number of parallax images can be the quotient of the width of the display divided by the width of the raster unit. In some embodiments, the image generation module 237 may also obtain the pixel pitch of the display and the grating width of the raster film, and calculate the quotient of the pixel pitch divided by the grating width as the number of parallax images. The aforementioned pixel pitch and grating width may be recorded in the characteristic parameters, and the pixel pitch may also be calculated based on the display resolution, diagonal size, and width in the characteristic parameters. For example, the pixel pitch may be the quotient of the display width divided by the display's horizontal resolution, wherein the display width may also be calculated by dividing the display's diagonal size by the square root of 1 + (vertical resolution / horizontal resolution).

[0097] The image generation module 237 is responsible for generating multi-view images based on the final depth image generated by the depth enhancement module 235 and the determined raster characteristics of the stereoscopic display device 250. The multi-view images generated by the image generation module 237 include display images from multiple different perspectives. More specifically, the image generation module 237 can calculate disparity based on the depth map, and generate left and right view images (Stereo Pair) by offsetting pixels in the image according to the depth information through disparity mapping (Depth-based Image Warping), and further generate multi-view images with the same number of display images as the number of disparity images.

[0098] When the image generation module 237 can also generate the multi-view image, it interleaves the subpixels of each pixel in each displayed image to generate the multi-view image, such as... Figure 4 As shown, this improves the clarity and 3D visual effect of multi-view images. The aforementioned subpixels are typically the smallest light-emitting units that emit a single color (R / G / B) from the three primary colors.

[0099] The image generation module 237 can also generate conversion completion information corresponding to the same stereoscopic display device 250 after the multi-view images provided to the stereoscopic display device 250 are generated. In some embodiments, the conversion completion information generated by the image generation module 239 includes image recognition data of the multi-view images and device identification data of the stereoscopic display device 250 corresponding to the multi-view images. The image recognition data may be a file name, and the device identification data may be the factory serial number of the stereoscopic display device 250 or the serial number of a specific hardware component, etc., but the present invention is not limited to the above.

[0100] The stereoscopic display device 250 is responsible for displaying the multi-view images generated by the conversion server 230 in a three-dimensional manner. In some embodiments, the processor of the stereoscopic display device 250 may include modules such as a device transmission module 251 and a device display module 253.

[0101] The device transmission module 251 is responsible for downloading multi-view images generated by the conversion server 230 through the communication interface of the stereoscopic display device 250. The device transmission module 251 can connect to the conversion server 230 to download the multi-view images generated by the conversion server, or it can connect to the cloud server 220 to download the multi-view images transmitted by the conversion server 230 and stored on the cloud server 220. This invention has no particular limitations. In some embodiments, the device transmission module 251 can generate image query requests containing device identification data at regular intervals or at fixed intervals, and can transmit the generated image query requests to the cloud server 220 to check whether there are converted multi-view images. If not, the device transmission module 251 waits to execute again next time; while if the conversion server 230 has generated multi-view images or the cloud server 220 has stored multi-view images, the device transmission module 251 can download the multi-view images from the conversion server 230 or the cloud server 220.

[0102] The device display module 253 can display the multi-view images received by the device transmission module 251. In some embodiments, the device display module 253 can also convert the multi-view images received by the device transmission module 251 into multiple display images, and then display the converted display images.

[0103] The display module 253 of the device includes a grating film, which contains a grating and a plurality of grating units. Generally, the size of the grating film is the same as the size of the display or touch screen of the output unit of the stereoscopic display device, but the present invention is not limited thereto.

[0104] The device display module 253 can display multi-view images downloaded by the device transmission module 251, and through the grating superposition in the grating film, the multi-view images are displayed to present corresponding display images in multiple different viewpoints through optical principles, thereby creating a layered image by highlighting and deepening the image through visual illusion.

[0105] The device display module 253 can also select whether to display multi-view images according to the display mode received by the device transmission module 251. More specifically, when the display mode is a planar image, the device display module 253 can display a planar target image, and at this time, the grating can be ignored; while when the display mode is a stereoscopic image, the display device 253 can display multi-view images and present a stereoscopic image by superimposing the multi-view images through the grating.

[0106] The operating system and method of the present invention will then be explained using an embodiment, and please refer to [reference needed]. Figure 3A The flowchart of the method for converting two-dimensional images into multi-view images according to user level proposed in this invention.

[0107] First, the client 210 can select a target image and set the device information of the stereoscopic display device 250, and can transmit the selected target image and the set device information to the cloud server 220 (step 310). In this embodiment, it is assumed that the target image is a two-dimensional image, and the device information includes the model and device identification data of the stereoscopic display device 250.

[0108] After the cloud transmission module 221 of the cloud server 220 receives the target image and device information transmitted by the client 210, the parameter generation module 227 of the cloud server 220 can obtain the user level of the client's user and generate conversion parameters based on the obtained user level (step 321). In this embodiment, it is assumed that the user level includes non-member, basic member, advanced member, etc. If the client 210 transmits user identification data after connecting to the cloud server 220 (or the cloud transmission module 221 obtains user identification data based on the login data transmitted by the client 210), the parameter generation module 227 can query the user level based on the user identification data of the client 210's user. If the client 210 does not transmit user identification data, the parameter generation module 227 can obtain the non-member user level.

[0109] Similarly, after the cloud transmission module 221 of the cloud server 220 receives the target image and the device information of the stereoscopic display device 250 transmitted by the client 210, the parameter generation module 227 of the cloud server 220 can convert the device information into characteristic parameters of the stereoscopic display device 250. After the parameter generation module 227 generates the conversion parameters, the cloud transmission module 221 can transmit the received target image and the characteristic parameters and conversion parameters generated by the parameter generation module 227 to the conversion server 230 (step 325).

[0110] It should be noted that there is no restriction on the order in which the parameter generation module 227 of the cloud server 220 generates conversion parameters based on the user level (step 321) and the parameter generation module 227 converts device information into characteristic parameters (step 325). In other words, the parameter generation module 227 can first convert the device information into characteristic parameters and then generate conversion parameters based on the user level, so that the cloud transmission module 221 of the cloud server 220 can transmit the target image, characteristic parameters and conversion parameters to the conversion server 230.

[0111] After the service transmission module 231 of the conversion server 230 receives the target image, feature parameters, and conversion parameters transmitted by the cloud server 220, the depth generation module 233 of the conversion server 230 can use a deep learning model to generate an initial depth map of the target image received by the service transmission module 231 (step 340). In this embodiment, it is assumed that the deep learning model used by the conversion server can be a machine learning model capable of depth estimation disclosed in the paper "Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-datasetTransfer" published on March 1, 2022 in "IEEE Transactions on Pattern Analysis and Machine Intelligence (Volume:44, Issue:3)" to generate the initial depth map.

[0112] After the depth generation module 233 of the conversion server 230 generates the initial depth map, the depth enhancement module 235 of the conversion server 230 can selectively perform image enhancement calculations on the initial depth map generated by the depth generation module 233 to generate the final depth image based on the conversion parameters received by the service transmission module 231 of the conversion server 230 (step 350). In this embodiment, assuming that the conversion parameters indicate that all image enhancement calculations can be performed, the depth enhancement module 235 can perform the following: Figure 3BAs shown in the flowchart, the depth enhancement module 235 can sequentially perform image enhancement calculations of brightness expansion and equalization on the initial depth map to generate a first intermediate image that can obtain more stereoscopic depth details and increase image color contrast (step 351), and can perform low-pass filtering on the first intermediate image to generate a smoothed second intermediate image (step 353), and can perform visual depth constraint linear transformation on the second intermediate image to generate a final depth image that conforms to the rules of visual perception (step 355).

[0113] After the depth enhancement module 235 of the conversion server 230 generates the final depth image, the image generation module 239 of the conversion server 230 can determine the raster characteristics of the stereoscopic display device 250 based on the characteristic parameters received by the service transmission module 231 of the conversion server 230 (step 360). In this embodiment, it is assumed that the raster characteristics of the stereoscopic display device 250 determined by the image generation module 239 include the number of parallax images, the parallax range, and the viewing distance.

[0114] After the depth enhancement module 235 of the conversion server 230 generates the final depth image, the image generation module 237 of the conversion server 230 can generate a multi-view image (step 370) based on the final depth image generated by the depth enhancement module 235, which conforms to the characteristics of the raster film of the stereoscopic display device 250 and displays multiple different viewing angles. In this embodiment, it is assumed that the image generation module 237 can interleave the subpixels of each pixel in each display image to generate a multi-view image with subpixel-level interleaving.

[0115] After the image generation module 230 of the conversion server 230 generates multi-view images, the device transmission module 251 of the stereoscopic display device 250 can download the multi-view images generated by the conversion server 230 (step 380). In this embodiment, it is assumed that... Figure 3C As shown in the process, after the image generation module 237 of the conversion server 230 generates multi-view images, conversion completion information can be generated. The service transmission module 231 of the conversion server 230 can transmit the conversion completion information and multi-view images to the cloud server 220 for storage (step 381). The image storage and retrieval module 229 of the cloud server 220 can provide the device transmission module 251 of the stereoscopic display device 250 with a query to see if there is corresponding conversion completion information (step 383). If so, the device transmission module 251 can download the multi-view images generated by the conversion server 230 (step 385).

[0116] After the device transmission module 251 of the stereoscopic display device 250 downloads the multi-view images generated by the conversion server 230, the device display module 253 of the stereoscopic display device 250 can display the multi-view images, and can use the grating of the device display module 253 to superimpose the displayed multi-view images, thereby presenting a display image corresponding to each viewpoint in the multi-view images from different viewpoints through optical principles (step 390).

[0117] Thus, through this invention, two-dimensional planar images can be converted into multi-view images that can be displayed in stereo according to the user level, and three-dimensional stereo images can be displayed by overlaying multi-view images through grating.

[0118] In the above embodiments, before the parameter generation module 227 of the cloud server 220 generates conversion parameters according to the user level (step 321) and before the parameter generation module 227 converts the device information into characteristic parameters (step 325), it is possible to further perform the following steps: Figure 3D As shown in the process, the client 210 selects a payment plan and transmits the payment plan to the cloud server 220 (step 301). After the cloud transmission module 221 of the cloud server 220 receives the payment plan, the payment processing module 223 of the cloud server 220 can provide the client 210 with the payment operation corresponding to the selected payment plan (step 303). After the client 210 completes the payment operation, the level setting module 225 of the cloud server 220 can set the user level of the client 210 according to the payment plan (step 305).

[0119] In summary, the difference between this invention and existing technologies lies in the fact that the cloud server generates characteristic parameters based on the device information transmitted by the client and conversion parameters based on the client's user level. The conversion server then selects an image enhancement calculation based on the initial depth map of the target image transmitted by the client to generate a final depth image. Based on the characteristic parameters, it determines the grating film characteristics of the stereoscopic display device and generates multi-view images that conform to the grating film characteristics of the stereoscopic display device based on the final depth image. The stereoscopic display device then downloads and displays these multi-view images. This allows the stereoscopic display device to present stereoscopic images by displaying various images from different perspectives through optical principles after the grating overlays the multi-view images. This technology solves the problem of existing technologies where image conversion services require financial investment and are not suitable for free provision. It achieves the technical effect of providing multi-view images with different comfort levels and resolutions of stereoscopic images through different payment options.

[0120] Furthermore, the method of converting two-dimensional images into multi-view images according to user level of the present invention can be implemented in hardware, software or a combination of hardware and software, and can also be implemented in a centralized manner in a computer system or in a decentralized manner with different components distributed among several interconnected computer systems.

[0121] While the embodiments disclosed in this invention are as described above, the content is not intended to directly limit the scope of patent protection for this invention. Any modifications or alterations to the form and details of the implementation of this invention made by those skilled in the art without departing from the spirit and scope disclosed herein shall fall within the scope of patent protection for this invention. The scope of patent protection for this invention shall still be determined by the scope defined in the appended claims.

Claims

1. A method for converting two-dimensional images into multi-view images according to user hierarchy, characterized in that, The method includes at least the following steps: The client selects a target image and sets the device information of the stereoscopic display device, and transmits the target image and the device information to the cloud server. The target image is a two-dimensional image. The cloud server obtains the user level of the client and generates conversion parameters based on the user level; The cloud server converts the device information into characteristic parameters of the stereoscopic display device, and transmits the target image, the characteristic parameters, and the conversion parameters to the conversion server. The characteristic parameters include the number of parallax images. The conversion server uses a deep learning model to generate an initial depth map of the target image; The conversion server selects to selectively perform at least one image enhancement calculation on the initial depth map based on the conversion parameters to generate a final depth image; The conversion server determines the characteristics of the raster film of the stereoscopic display device based on the characteristic parameters; The conversion server generates a multi-view image that matches the characteristics of the raster model based on the final depth image, and the multi-view image includes display images from multiple different perspectives. The stereoscopic display device downloads the multi-view image after the conversion server generates the multi-view image; and The stereoscopic display device displays the multi-view images and uses the grating of the grating film of the stereoscopic display device to superimpose the multi-view images so as to present the display images corresponding to each of the different viewpoints in the multi-view images through optical principles.

2. The method for converting two-dimensional images into multi-view images according to user hierarchy as described in claim 1, characterized in that, The step of the stereoscopic display device downloading the multi-view image after the conversion server generates the multi-view image further includes the conversion server generating conversion completion information corresponding to the stereoscopic display device after generating the multi-view image and transmitting the conversion completion information and the multi-view image to the cloud server. The cloud server provides the stereoscopic display device with the step of downloading the multi-view image when the conversion completion information exists.

3. The method for converting two-dimensional images into multi-view images according to user hierarchy as described in claim 1, characterized in that, The step of the conversion server selectively performing at least one image enhancement calculation on the initial depth map according to the conversion parameters to generate the final depth image further includes the conversion server performing a linear transformation on each brightness range of the initial depth map to generate a first intermediate image, filtering the first intermediate image to generate a second intermediate image, and performing a visual depth constraint linear transformation on the second intermediate image to generate the final depth image.

4. The method for converting two-dimensional images into multi-view images according to user hierarchy as described in claim 1, characterized in that, Before the step of obtaining the user level of the client from the cloud server, the method further includes the step of the client selecting a payment plan and transmitting the payment plan to the cloud server, and the cloud server setting the user level of the client according to the payment plan after the client completes the payment operation corresponding to the payment plan.

5. The method for converting two-dimensional images into multi-view images according to user hierarchy as described in claim 1, characterized in that, The step of the conversion server generating the multi-view image that matches the raster film characteristics of the stereoscopic display device based on the final depth image characteristic parameters further includes the step of the conversion server interleaving the sub-pixels of each pixel in each of the displayed images to generate the multi-view image.

6. A system for converting two-dimensional images into multi-view images according to user hierarchy, characterized in that, The system includes at least: The client is used to select a target image and set device information, wherein the target image is a two-dimensional image; A cloud server is used to receive the target image and the device information, obtain the user level of the client, and generate conversion parameters based on the user level, and convert the device information into characteristic parameters, the characteristic parameters including the number of disparity images; A conversion server is configured to receive the target image, the characteristic parameters, and the conversion parameters; generate an initial depth map of the target image using a deep learning model; selectively perform at least one image enhancement operation on the initial depth map based on the conversion parameters to generate a final depth image; determine the grating film characteristics of the stereoscopic display device based on the characteristic parameters; and generate a multi-view image matching the grating film characteristics based on the final depth image, wherein the multi-view image includes multiple display images from different perspectives. A stereoscopic display device, corresponding to the device information, includes a grating film, the grating film containing a grating and possessing the characteristics of a grating film. The stereoscopic display device is used to download the multi-view image after the conversion server generates the multi-view image, and to display the multi-view image. The grating film is used to superimpose the multi-view image to present the display image corresponding to each of the different viewpoints in the multi-view image through optical principles.

7. The system for converting two-dimensional images into multi-view images according to user hierarchy as described in claim 6, characterized in that, The conversion server is also used to generate conversion completion information corresponding to the stereoscopic display device after generating the multi-view image, and to transmit the conversion completion information and the multi-view image to the cloud server. The cloud server is also used to provide the stereoscopic display device with the option to download the multi-view image when the conversion completion information exists.

8. The system for converting two-dimensional images into multi-view images according to user hierarchy as described in claim 6, characterized in that, The conversion server performs linear conversion on each brightness range of the initial depth map to generate a first intermediate image, filters the first intermediate image to generate a second intermediate image, and performs visual depth constraint linear conversion on the second intermediate image to generate the final depth image.

9. The system for converting two-dimensional images into multi-view images according to user hierarchy as described in claim 6, characterized in that, The client is also used to select a payment plan and transmit the payment plan to the cloud server. The cloud server is also used to set the user level of the client according to the payment plan after the client completes the payment operation corresponding to the payment plan.

10. The system for converting two-dimensional images into multi-view images according to user hierarchy as described in claim 6, characterized in that, The conversion server is also used to interleave the subpixels of each pixel in each of the displayed images to generate the multi-view images.

Images