Image processing apparatus, image processing method, learning method of neural network, texture mapping system, texture mapping method, and edge device
The image processing device and texture mapping system enhance pixel count and definition using edge enhancement and neural networks, addressing the challenges of image quality and data transmission.
Patent Information
- Application Number
- JP2023077885
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-21
AI Technical Summary
Existing image processing technologies face challenges in increasing pixel count without degrading image definition and reducing data transmission in texture mapping processes.
An image processing device and method that includes a pixel count increasing unit and definition enhancement section, utilizing edge enhancement filters and noise removal filters, followed by neural network processing to improve image definition. In texture mapping, a cloud server transmits low-pixel images to an edge device, which generates high-pixel images using neural networks, reducing transmission data.
The solution effectively increases pixel count without degrading image definition and minimizes data transmission, enhancing image quality and reducing network load.
Smart Images

Figure 2024162406000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an image processing device, an image processing method, a neural network learning method, a texture mapping system, a texture mapping method, and an edge device. [Background technology]
[0002] (1) Non-Patent Document 1 discloses a super-resolution technology using machine learning. In detail, the document discloses that a high-pixel-count image is generated by increasing the number of pixels of a low-pixel-count image by bicubic interpolation, and the high-pixel-count image is subjected to a convolutional neural network to improve the sense of detail.
[0003] (2) In texture mapping, a technique called mipmap is known. With this technique, multiple texture images with different numbers of pixels are prepared for the same texture content, and texture mapping is performed using the appropriate texture image. As an example, multiple texture images are sent from a cloud server to an edge device, and texture mapping is performed on the edge device. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang "Image Super-Resolution Using Deep Convolutional Networks" 6 / 2014, ECCV 2014, (1309) [Non-Patent Document 2] https: / / imagingsolution.net / imaging / bilateralfilter / Summary of the Invention [Problem to be solved by the invention]
[0005] The objectives of the present disclosure are (1) to increase the number of pixels in an image without compromising detail as much as possible, and / or (2) to reduce the amount of data transmitted as much as possible in texture mapping. [Means for solving the problem]
[0006] According to one aspect of the present disclosure, [1] An image processing device that generates an image to be input to a neural network unit that improves the definition of the image, a pixel count increasing processor that increases the number of pixels of a first image to be processed to generate a second image; and a definition enhancing section that enhances the definition of the second image to generate an image to be input to the neural network section.
[0007] [2] In the image processing device according to [1], The definition improving unit includes: performing edge enhancement on the second image to generate a third image; An image to be input to the neural network unit may be generated by removing noise from the third image while preserving edges.
[0008] [3] In the image processing device according to [1], The definition improving unit includes: an edge enhancement filter that performs edge enhancement on the second image to generate a third image; The image processing device may further include an edge-preserving noise removal filter that removes noise from the third image while preserving edges to generate an image to be input to the neural network unit.
[0009] [4] In the image processing device according to [3], The edge enhancement filter may be a kernel-based filter.
[0010] [5] In the image processing device according to [3] or [4], The edge-preserving denoising filter may be a bilateral filter.
[0011] According to one aspect of the present disclosure, [6] [1] to [5], and an image processing device according to any one of the above. and a neural network unit that improves the sense of detail of the image generated by the image processing device.
[0012] According to one aspect of the present disclosure, [7] An image processing method for generating an image to be input to a neural network unit that improves the definition of the image, comprising: generating a second image by increasing the number of pixels of a first image to be processed; and enhancing the definition of the second image to generate an image to be input to the neural network unit.
[0013] According to one aspect of the present disclosure, [8] generating a first image by reducing the number of pixels of the original image; increasing the number of pixels of the first image to generate a second image; enhancing the detail of the second image to generate a third image; and using the original image and the third image, training a neural network section that improves image definition is provided.
[0014] According to one aspect of the present disclosure, [9] generating a second image by increasing the number of pixels of a first image to be processed; enhancing the detail of the second image to generate a third image; and inputting the third image into a trained neural network for enhancing image detail to enhance the image detail.
[0015] According to one aspect of the present disclosure,
[10] Equipped with a cloud server and an edge device, the cloud server has a transmission unit configured to transmit one or more first texture images, the maximum number of pixels of which is a first number of pixels, to the edge device; The edge device includes: a receiving unit for receiving the one or more first texture images transmitted from the cloud server; a pixel count increasing processor that generates a second texture image having a second pixel count that is greater than the first pixel count, based on at least one of the one or more first texture images; A texture mapping system is provided having a texture mapping unit that performs texture mapping based on the second texture image.
[0016]
[11]
[10] , wherein the texture mapping system comprises: the pixel count increasing processor generates the second texture image using a neural network; The transmitter may transmit network parameters of the neural network to the edge device.
[0017]
[12]
[10] or
[11] , in a texture mapping system according to the present invention, The pixel count increasing processing unit includes: (1) generating a third texture image based on at least one of the one or more first texture images, the third texture image having a third number of pixels greater than the second number of pixels; or (2) A third texture image having the third number of pixels may be generated based on the second texture image generated based on at least one of the one or more first texture images.
[0018]
[13]
[10] or
[11] , in a texture mapping system according to the present invention, The pixel count increasing processing unit includes: Dividing the second texture image generated based on at least one of the one or more first texture images into a plurality of first divided images; increasing the number of pixels of each of the plurality of first divided images to generate a plurality of second divided images; The plurality of second divided images may be combined to generate a third texture image.
[0019]
[14]
[13] A texture mapping system according to the present invention, the number of pixels of the one or more first texture images is the first number of pixels; the number of pixels of the second texture image is the second number of pixels; the number of pixels of each of the first divided images is the first number of pixels; the number of pixels of each of the second divided images is the second number of pixels; the pixel count increasing processor includes a neural network unit that increases the pixel count of an image from the first pixel count to the second pixel count, The neural network unit includes: generating the second texture image based on at least one of the one or more first texture images; and The second divided images may be generated based on the first divided images, respectively.
[0020]
[15]
[13] or
[14] , The pixel count increasing processor may perform a smoothing process on a boundary between one of the second divided images and another of the second divided images when combining the plurality of second divided images.
[0021]
[16]
[10] to
[15] , wherein the texture mapping system comprises: the transmission unit transmits auxiliary information used in generating the second texture image to the edge device; The pixel count increasing processor may generate the second texture image based on at least one of the one or more first texture images and the auxiliary information.
[0022]
[17]
[16] A texture mapping system as described in The pixel count increasing processing unit includes: a first neural network unit that generates a first provisional texture image having the second number of pixels based on at least one of the one or more first texture images; The image processing device may further include a texture image generating unit that generates the second texture image based on the provisional texture image and the auxiliary information.
[0023]
[18]
[17] A texture mapping system as described in The cloud server includes: a pixel reduction processing unit that reduces the number of pixels of the original image having the second pixel number to generate a reduced pixel number image; a second neural network unit that generates a second provisional texture image based on the low pixel count image by the same process as that of the first neural network unit; The image processing device may further include an auxiliary information generating unit that generates the auxiliary information according to a difference between the original image and the second temporary texture image.
[0024]
[19]
[18] , in a texture mapping system according to the present invention, The auxiliary information generating section may compress the difference to generate the auxiliary information.
[0025]
[20]
[18] or
[19] , wherein the texture mapping system The amount of data required to transmit the auxiliary information to the edge device may be less than the amount of data required to transmit the original image to the edge device.
[0026] According to one aspect of the present disclosure, [twenty one] a cloud server transmitting one or more first texture images to the edge device, the first texture images having a maximum number of pixels being a first number of pixels; receiving, by an edge device, the one or more first texture images transmitted from the cloud server; generating, by the edge device, a second texture image based on at least one of the one or more first texture images, the second texture image having a second number of pixels greater than the first number of pixels; and performing texture mapping by the edge device based on the second texture image.
[0027] According to one aspect of the present disclosure, [twenty two] a receiving unit that receives one or more first texture images, the maximum number of pixels of which is a first number of pixels, transmitted from a cloud server; a pixel count increasing processor that generates a second texture image having a second pixel count that is greater than the first pixel count, based on at least one of the one or more first texture images; An edge device is provided, comprising: a texture mapping unit that performs texture mapping based on the second texture image.
[0028] According to one aspect of the present disclosure, [twenty three] receiving, by the edge device, one or more first texture images transmitted from a cloud server, the first texture images having a maximum number of pixels being a first number of pixels; generating, by the edge device, a second texture image based on at least one of the one or more first texture images, the second texture image having a second number of pixels greater than the first number of pixels; and performing texture mapping by the edge device based on the second texture image. [Brief description of the drawings]
[0029] [Figure 1] 1 is a block diagram showing a schematic configuration of an image processing system 100 according to a first embodiment. [Diagram 2] FIG. 4 is a diagram for explaining an example of an edge emphasis filter 121. [Diagram 3] 4 is a flowchart showing a procedure for performing learning in the neural network unit 2. [Figure 4] 4 is a flowchart showing a procedure for performing inference by the neural network unit 2. [Diagram 5] FIG. 11 is a block diagram showing a schematic configuration of a texture mapping system 200 according to a second embodiment. [Figure 6] 3A to 3C are diagrams for explaining the processing operation of the texture mapping system 200. [Figure 7A] 5A to 5C are diagrams for explaining an example of processing by a pixel count increasing processor 42. [Figure 7B] FIG. 7B is a schematic block diagram of a pixel count increasing processor 42 for performing the processing of FIG. 7A. [Figure 8A] 10A to 10C are diagrams for explaining another example of the process of the pixel count increasing processor 42. [Figure 8B] FIG. 8B is a schematic block diagram of a pixel count increasing processor 42 for performing the processing in FIG. 8A. [Figure 9A] FIG. 4 is a diagram for explaining a learning method of the neural network unit 423. [Figure 9B] 13 is a flowchart showing a procedure for performing learning in the neural network unit 423. [Figure 10A]FIG. 4 is a diagram for explaining a learning method of the neural network unit 424. [Figure 10B] 11 is a flowchart showing a procedure for performing learning in the neural network unit 424. [Figure 11A] 13 is a diagram for explaining still another example of the process of the pixel count increasing processor 42. FIG. [Figure 11B] FIG. 11B is a schematic block diagram of a pixel count increasing processor 42 for performing the processing of FIG. 11A. [Figure 12A] FIG. 11 is a block diagram showing a schematic configuration of a texture mapping system 201 according to a third embodiment. [Figure 12B] 13A to 13C are diagrams for explaining generation of auxiliary information performed by the cloud server 3 in the third embodiment. [Figure 12C] 13A to 13C are diagrams for explaining generation of a texture image using auxiliary information performed by the edge device 4 in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0030] Hereinafter, an embodiment of the present invention will be specifically described with reference to the drawings.
[0031] (First embodiment) In the first embodiment, the number of pixels of an image is increased without compromising the sense of detail as much as possible. Note that the image to be processed may be a moving image or a still image (including a still image that constitutes a moving image).
[0032] 1 is a block diagram showing a schematic configuration of an image processing system 100 according to the first embodiment. The image processing system 100 includes a preprocessing unit 1 (image processing device) and a neural network unit 2.
[0033] The pre-processing section 1 generates an image to be input to the neural network section 2, and includes a pixel count increasing section 11 and a definition improving section 12.
[0034] The pixel count increasing processor 11 increases the number of pixels of a low pixel count image (first image) to be processed, for example, by bicubic interpolation, to generate an image (second image) to be input to the definition improving processor 12. The number of pixels of the low pixel count image is, for example, 2048 pixels x 1024 pixels. The number of pixels of the image generated by the pixel count increasing processor 11 is, for example, 4096 pixels x 2048 pixels. However, these pixel counts are merely examples.
[0035] The definition improving section 12 improves the definition of the image generated by the pixel count increasing processing section 11, and generates an image to be input to the neural network section 2. Note that the number of pixels of the image input to the definition improving section 12 and the number of pixels of the image output from the definition improving section 12 may be different or the same. An example of the configuration of the definition improving section 12 will be specifically described below.
[0036] When the pixel count increasing processor 11 increases the number of pixels by bicubic interpolation, jaggies may appear on diagonal edges. Also, bicubic interpolation may generate a blurred image (losing high-frequency bands). Therefore, even if an image including such diagonal jaggies or a blurred image is input to the neural network unit 2, it is not necessarily possible to generate a high-pixel image with sufficient detail.
[0037] Therefore, in this embodiment, the definition enhancing section 12 performs processing to emphasize edges on the image generated by the pixel count increasing processing section 11. Furthermore, the definition enhancing section 12 removes noise while preserving edges from the image (third image) generated by this processing, and generates an image to be input to the neural network section 2. As an example of the configuration for this purpose, the definition enhancing section 12 has an edge emphasizing filter 121 and an edge-preserving noise removal filter 122.
[0038] The edge emphasis filter 121 performs edge emphasis on the image generated by the pixel count increasing processor 11 .
[0039] 2 is a diagram illustrating an example of the edge enhancement filter 121. As illustrated, the edge enhancement filter 121 may be a filter using a kernel. In this filter, a window of (2W+1) pixels in the horizontal direction and (2W+1) pixels in the vertical direction is set for the pixel to be processed (W=1 in the example of FIG. 2). Then, the edge enhancement filter 121 sets a value obtained by adding up values obtained by applying weighting coefficients to each pixel in the window as the value of the pixel to be processed after filtering.
[0040] Specifically, k in FIG. 2 is a positive number, and the weighting coefficient applied to the pixel to be processed is greater than 1. Also, the weighting coefficients applied to the pixels surrounding the pixel to be processed are negative. In the example shown in FIG. 2, the weighting coefficient applied to the pixel to be processed is 8*k / 9 greater than 1, and the weighting coefficients applied to the surrounding pixels are all -k / 8. The sum of the weighting coefficients is 1.
[0041] The larger the value of k, the more edges are emphasized, but the more noise is emphasized. Therefore, the value of k should be set appropriately, taking into consideration the type of image to be processed and whether the importance of edge preservation or noise removal is to be given.
[0042] 2, the edge-preserving noise removal filter 122 removes noise while preserving edges from the image generated by the edge enhancement filter 121 to generate an image to be input to the neural network unit 2. The edge-preserving noise removal filter 122 may be, for example, a bilateral filter.
[0043] Bilateral filters are similar to kernel filters in that a window of (2W+1) pixels in the horizontal direction and (2W+1) pixels in the vertical direction is set for the pixel to be processed, and the value obtained by adding up the values obtained by applying weighting coefficients to each pixel in the window is used as the value of the pixel to be processed after filtering. However, the weighting coefficients applied to the pixels surrounding the pixel to be processed are not constant.
[0044] Specifically, the weighting coefficient applied to the surrounding pixels of the target pixel is smaller the farther the pixel is from the target pixel. This is similar to a general Gaussian filter. Furthermore, the weighting coefficient applied to the surrounding pixels of the target pixel is larger the closer the pixel is to the target pixel.
[0045] More specifically, for a pixel (i,j) that is the i-th pixel from the left and the j-th pixel from the top in an image, if the value of the pixel to be processed before filtering is f(i,j) and the value of the pixel to be processed after filtering is g(i,j), then the pixel value g(i,j) after filtering is expressed by the following equation (1).
number
[0046] The following equation (2) in equation (1) is a term added to the Gaussian filter.
number
[0047] Here, σ1 is a parameter that controls the degree of application of the Gaussian filter, and σ2 is a parameter that controls the degree of application of the bilateral filter. The values of σ1 and σ2 may be set appropriately.
[0048] It should be noted that the above-described configuration of the fineness improving section 12 is merely an example. The fineness improving section 12 may perform any processing that improves the fineness of the image generated by the pixel count increasing processing section 11. In other words, when an image generated by the pixel count increasing processing section 11 is compared with an image generated by performing appropriate processing on this image, if the latter has improved fineness, then the processing section that performs the appropriate processing can be said to correspond to the fineness improving section 12.
[0049] The image output from the pre-processing unit 1 described above is input to the neural network unit 2, which further improves the sense of detail. More specifically, the neural network unit 2 processes a high-pixel-count image having a greater number of pixels than the low-pixel-count image input to the pre-processing unit 1, and generates and outputs an image with improved sense of detail. The number of pixels of the image input to the neural network unit 2 and the number of pixels of the image output from the neural network unit 2 may be the same or different.
[0050] The image processing system 100 in the figure can be applied to both the learning and inference of the neural network unit 2.
[0051] 3 is a flowchart showing a procedure for learning the neural network unit 2. Here, it is assumed that there are a sufficiently large number of original images (Ground Truth: GT) in order to obtain effective learning results.
[0052] First, the number of pixels in the original image is reduced by an appropriate method (e.g., downsampling) to generate a low-pixel-count image (step S1). Next, the pixel-count increasing processor 11 in Fig. 1 increases the number of pixels in the low-pixel-count image to generate a high-pixel-count image (step S2). Then, the fineness improving unit 12 improves the fineness of the high-pixel-count image (step S3).
[0053] A large number of pairs of the high-pixel-count images thus generated and the original images are prepared. Then, the high-pixel-count images are used as inputs and the original images are used as correct answer data to train the neural network unit 2 (step S4). As a specific example, the difference (e.g., mean square error) between the image obtained by processing the high-pixel-count image by the neural network unit 2 and the original image is used as a loss function, and the network parameters (weights) of the neural network unit 2 are calculated by training so that this loss function is as small as possible (i.e., so that the sense of detail is not lost as much as possible).
[0054] In this manner, the neural network section 2 learns, and becomes able to improve the sense of detail of an image with a high pixel count.
[0055] 4 is a flowchart showing the procedure for performing inference by the neural network unit 2. Here, it is assumed that the neural network unit 2 has already been trained.
[0056] 1 increases the number of pixels of a low-pixel image to be processed to generate a high-pixel image (step S11). Then, the fineness improvement unit 12 improves the fineness of the high-pixel image (step S12). Next, the image with improved fineness is input to the trained neural network unit 2, and the neural network unit 2 generates an image with even more improved fineness (step S13).
[0057] In this way, in the first embodiment, the image obtained by performing the process of improving the fineness by the fineness improving unit 12 is input to the neural network unit 2. Therefore, it is possible to increase the number of pixels in the image without compromising the fineness as much as possible.
[0058] Second embodiment The second embodiment is intended to reduce the amount of data transmitted as much as possible in texture mapping.
[0059] Texture mapping is a computer graphics technique that maps a texture image onto an object made up of polygon data. The mipmap technique used in texture mapping prepares multiple texture images with different numbers of pixels for a specific texture. Then, the texture image with the appropriate number of pixels is selectively pasted depending on the distance from the object to which it is to be pasted. This increases the drawing speed and prevents aliasing.
[0060] A possible configuration is to store multiple texture images with different numbers of pixels in a cloud server, transmit them to an edge device (e.g., a smartphone or tablet terminal), and perform texture mapping on the edge device using the texture image with an appropriate number of pixels. However, such a configuration has the problem that a large amount of data is transmitted from the cloud server to the edge device, resulting in a large amount of data transmission and a heavy load on the network. Another problem is that a large number of texture images must be stored in the cloud server, which requires the cloud server to have a large-capacity storage medium.
[0061] Therefore, in this embodiment, only a texture image with a small number of pixels is transmitted from the cloud server to the edge device, and a texture image with a large number of pixels is generated at the edge device. This will be described in detail below.
[0062] 5 is a block diagram showing a schematic configuration of a texture mapping system 200 according to the second embodiment. The texture mapping system 200 includes a cloud server 3 and an edge device 4.
[0063] The cloud server 3 has a storage means 31 and a transmission unit 32. The storage means 31 holds a plurality of (hereinafter referred to as (m-2)) texture images (first to (m-2)th texture images) each having a different number of pixels. The transmission unit 32 transmits these texture images to the edge device 4.
[0064] 6 is a diagram for explaining the processing operation of the texture mapping system 200. In the illustrated example, the number of pixels of the first texture image stored in the storage means 31 of the cloud server 3 is 1 pixel×1 pixel, the number of pixels of the second texture image is 2×2 pixels, and the number of pixels of the (m-2)th texture image is n / 4 pixels×n / 4 pixels, and these are transmitted to the edge device 4. In other words, the maximum number of pixels of the texture image transmitted to the edge device 4 by the transmission unit 32 is n / 4×n / 4 pixels. However, there is no particular limit to the number of pixels, and for example, the number of pixels in the horizontal direction and the number of pixels in the vertical direction may be different.
[0065] Returning to FIG. 5, the edge device 4 includes a receiving unit 41, a pixel count increasing processing unit 42, a texture mapping unit 43, and a storage unit 44.
[0066] The receiving unit 41 receives the texture transmitted from the cloud server 3, and stores it in the storage means 44. In the above example, the first to (m-2)th texture images are received by the receiving unit 41, and stored in the storage means 44.
[0067] Based on the received texture image, pixel count increasing processor 42 generates a texture image having a pixel count that is not transmitted from cloud server 3, and stores the generated texture image in storage means 44. In the above example, pixel count increasing processor 42 generates an (m-1)th texture image having a pixel count of n / 2 pixels×n / 2 pixels, and an mth texture image having a pixel count of n×n pixels, and stores the generated texture image in storage means 44 (see FIG. 6).
[0068] The pixel count increasing processor 42 may be configured to include a neural network. The pixel count increasing processor 42 desirably performs at least a process to increase the number of pixels, and also a process to improve the sense of detail. The pixel count increasing processor 42 may include, for example, the image processing system 100 described in the first embodiment, or may be one described in Non-Patent Document 1. A specific example of the processing by the pixel count increasing processor 42 will be described later.
[0069] The texture mapping unit 43 performs texture mapping based on the texture image stored in the storage means 44. As a specific example, the texture mapping unit 43 selects and pastes a texture image having an appropriate number of pixels according to the distance from the object to be pasted, from among the multiple texture images stored in the storage means 44. The selected texture image may be one received from the cloud server 3, or may be one generated by the pixel count increasing processor 42.
[0070] There is no limit to the number of texture images transmitted from the cloud server 3 to the edge device 4, as long as at least one texture image is transmitted. There is also no limit to the number of texture images generated by the edge device 4, as long as at least one texture image having a larger number of pixels than the maximum number of pixels of a received texture image is generated.
[0071] A specific example of the process in which the pixel count increasing processor 42 generates the (m-1)th texture image and the mth texture image from the (m-2)th texture image will be described below.
[0072] Fig. 7A is a diagram for explaining an example of the processing of the pixel number increasing processor 42. And Fig. 7B is a schematic block diagram of the pixel number increasing processor 42 for performing the processing of Fig. 7A.
[0073] In this example, as shown in FIG. 7A, a neural network is applied to the (m-2)th texture image, which has a pixel count of n / 4×n / 4, to generate the (m-1)th texture image, which has a pixel count of n / 2×n / 2, and another neural network is applied to the (m-2)th texture image, to generate the mth texture image, which has a pixel count of n×n.
[0074] As an example of the configuration for this purpose, as shown in Fig. 7B, pixel count increasing processing section 42 has neural network sections 421 and 422. Although not shown, neural network sections 421 and 422 may be composed of a processing section that increases the number of pixels of an input image by bicubic interpolation or the like, and a neural network. The same applies to the neural network section described later.
[0075] The (m-2)th texture image having the number of pixels of n / 4×n / 4 is input to the neural network unit 421. Then, the neural network unit 421 doubles the number of pixels of the (m-2)th texture image in both the horizontal and vertical directions to generate the (m-1)th texture image having the number of pixels of n / 2×n / 2.
[0076] The (m-2)th texture image, the number of pixels of which is n / 4×n / 4, is input to the neural network unit 422. Then, the neural network unit 422 multiplies the number of pixels of the (m-2)th texture image by four in both the horizontal and vertical directions to generate the mth texture image, the number of pixels of which is n×n.
[0077] In this case, the network structures and the numbers of network parameters of neural network units 421 and 422 may be the same or different. However, since neural network unit 421 doubles the number of pixels and neural network unit 422 quadruples the number of pixels, learning is performed separately. Therefore, the values of the network parameters are different from each other.
[0078] The network parameters may be pre-stored in the storage means 44 of the edge device 4, or may be transmitted from the transmission unit 32 of the cloud server 3. For example, the method described in Fig. 3 may be applied to the learning of the neural network units 421 and 422. This also applies to each neural network unit described later.
[0079] Fig. 8A is a diagram for explaining another example of the process of the pixel number increasing processor 42. And Fig. 8B is a schematic block diagram of the pixel number increasing processor 42 for performing the process of Fig. 8A.
[0080] In this example, as shown in FIG. 8A, a neural network is applied to the (m-2)th texture image, which has a pixel count of n / 4×n / 4, to generate the (m-1)th texture image, which has a pixel count of n / 2×n / 2, and then another neural network is applied to the generated (m-1)th texture image to generate the mth texture image, which has a pixel count of n×n.
[0081] As an example of the configuration for this purpose, pixel count increasing processor 42 has neural network sections 423 and 424 as shown in FIG. 8B.
[0082] The (m-2)th texture image having the number of pixels of n / 4×n / 4 is input to the neural network unit 423. Then, the neural network unit 421 doubles the number of pixels of the (m-2)th texture image in both the horizontal and vertical directions to generate the (m-1)th texture image having the number of pixels of n / 2×n / 2.
[0083] The (m-1)th texture image having n / 2×n / 2 pixels generated by the neural network unit 423 is input to the neural network unit 424. Then, the neural network unit 424 doubles the number of pixels of the (m-1)th texture image in both the horizontal and vertical directions to generate an mth texture image having n×n pixels.
[0084] A suitable learning method for the neural network units 423 and 424 will now be described. Fig. 9A is a diagram for explaining a learning method of neural network unit 423. Fig. 9B is a flowchart showing a procedure for performing learning in neural network unit 423.
[0085] First, the number of pixels of an original image (GT) I1, which has n×n pixels, is reduced by an appropriate method (e.g., downsampling) to generate an image I2, which has n / 2×n / 2 pixels (step S21). Furthermore, the number of pixels of the image I2 is reduced by an appropriate method (e.g., downsampling) to generate an image I3, which has n / 4×n / 4 pixels (step S22).
[0086] Many pairs of images I2 and I3, each having n / 2×n / 2 pixels and n / 4×n / 4 pixels, are prepared in this way. The latter is used as an input and the former as correct answer data to train the neural network unit 423 (step S23). As a specific example, the difference (e.g., mean square error) between image I2 and image I4, which has n / 2×n / 2 pixels and is obtained by processing image I3 by the neural network unit 423, is used as a loss function, and network parameters (weights) of the neural network unit 423 are calculated by training so that this loss function is as small as possible.
[0087] Fig. 10A is a diagram for explaining a learning method of neural network unit 424. Fig. 10B is a flowchart showing a procedure for learning neural network unit 424. It is assumed that learning of neural network unit 423 has already been completed as described above.
[0088] First, the number of pixels of an original image (GT) I1, which has n×n pixels, is reduced by an appropriate method (e.g., downsampling) to generate an image I2, which has n / 2×n / 2 pixels (step S31). Furthermore, the number of pixels of the image I2 is reduced by an appropriate method (e.g., downsampling) to generate an image I3, which has n / 4×n / 4 pixels (step S32). The above is the same as steps S21 and S22 in FIG. 9B.
[0089] Next, the trained neural network unit 423 generates an image I4 having n / 2×n / 2 pixels from the image I3 (step S33).
[0090] A large number of pairs of image I4, which has n / 2×n / 2 pixels and is generated in this way, and original image I1 are prepared. Then, the former is used as input and the latter as correct answer data, and the neural network unit 424 is trained (step S34). As a specific example, the difference (e.g., mean square error) between image I5, which has n / 2×n / 2 pixels and is obtained by processing image I4 by the neural network unit 424, and the original image I1 is used as a loss function, and the network parameters (weights) of the neural network unit 424 are calculated by training so that this loss function is as small as possible.
[0091] According to this method, the neural network unit 423 is trained first, and the neural network unit 424 can then be trained by going back to the original image. This makes it possible to produce a texture image with higher image quality.
[0092] Fig. 11A is a diagram for explaining yet another example of the process of the pixel number increasing processor 42. And Fig. 11B is a schematic block diagram of the pixel number increasing processor 42 for performing the process of Fig. 11A.
[0093] In this example, as shown in FIG. 11A, a neural network is applied to the (m-2)th texture image p1, which has a pixel count of n / 4×n / 4, to generate the (m-1)th texture image p2, which has a pixel count of n / 2×n / 2.
[0094] The (m-1)th texture image p2 is divided into four (specifically, divided into two equal parts vertically and horizontally) to generate four divided images p21 to p22 with n / 4×n / 4 pixels. A neural network is applied to each of the divided images p21 to p24 to generate four divided images p31 to p34 with n / 2×n / 2 pixels. The divided images p31 to p34 are then combined (pasted together) and smoothed as necessary to generate the mth texture image p3 with n×n pixels.
[0095] As an example of the configuration for this purpose, pixel count increasing processor 42 has a neural network section 425, a dividing section 426, and a combining section 427, as shown in FIG. 11B.
[0096] The (m-2)th texture image p1 and the divided images p21 to p24 from the dividing unit 426 are input in this order to the neural network unit 425. The number of pixels of each of these images is n / 4×n / 4. The neural network unit 425 then increases the number of pixels of the input images, and outputs an image with the number of pixels of n / 2×n / 2.
[0097] Specifically, when the (m-2)th texture image is input, the neural network unit 425 enhances the pixel count to generate the (m-1)th texture image p2, stores it in the storage means 44 (FIG. 5), and inputs it to the division unit 426. In addition, when the divided images p21 to p24 are input, the neural network unit 425 enhances the pixel count to generate divided images p31 to p34, respectively, and inputs them to the combination unit 427.
[0098] The (m-1)th texture image p2 output from the neural network unit 425 is input to the division unit 426. Then, the division unit 426 divides the (m-1)th texture image p2, which has n / 2×n / 2 pixels, into four to generate divided images p21 to p24, each having n / 4×n / 4 pixels.
[0099] The combination unit 427 receives the divided images p31 to p34, each having n / 2×n / 2 pixels, output from the neural network unit 425. The combination unit 427 then combines these images to generate the m-th texture image p3, each having n×n pixels. It is preferable that the combination unit 427 applies, for example, a smoothing filter to pixels near the boundaries between the divided images p31 to p34 so that discontinuities at the boundaries are not noticeable.
[0100] With this configuration, the neural network unit 425 performs pixel-enhancing processing on both the (m-2)th texture image p1 and the divided images p21 to p24. Therefore, the neural network unit 425 needs to hold only one type of network parameter for generating an image with n / 2×n / 2 pixels from an image with n / 4×n / 4 pixels. Therefore, the capacity of the network parameters to be transmitted from the cloud server 3 or to be held in the edge device 4 can be reduced.
[0101] As described above, in the second embodiment, the first to (m-2)th texture images are transmitted from the cloud server 3 to the edge device 4, and the (m-1)th and mth texture images with a higher number of pixels are generated at the edge device 4. Since there is no need to transmit the (m-1)th and mth texture images from the cloud server 3 to the edge device 4, it is possible to reduce the amount of data transmitted from the cloud server 3 to the edge device 4. In addition, there is no need to store texture images with a high number of pixels in the cloud server 3, so the capacity of the storage means 31 of the cloud server 3 can be reduced.
[0102] Third embodiment The third embodiment described next is a modification of the second embodiment. The following description will focus on the differences from the second embodiment.
[0103] In this embodiment, the transmitting unit 32 of the cloud server 3 transmits auxiliary information used for generating a texture image in the edge device 4 to the edge device 4. Then, the pixel count increasing processing unit 42 of the edge device 4 generates a texture image with a number of pixels greater than the maximum number of pixels of the texture image transmitted from the cloud server 3, based on the texture image and auxiliary information transmitted from the cloud server 3. Specific examples of the auxiliary information will be described below.
[0104] Fig. 12A is a block diagram showing a schematic configuration of a texture mapping system 201 according to the third embodiment. Fig. 12B is a diagram explaining the generation of auxiliary information performed by the cloud server 3 in the third embodiment. Fig. 12C is a diagram explaining the generation of a texture image using auxiliary information performed by the edge device 4 in the third embodiment.
[0105] The cloud server 3 further includes a pixel reduction processing unit 33, a neural network unit 34, and an auxiliary information generating unit 35.
[0106] 6, a texture image q1 having n / 2×n / 2 pixels as an original image is stored in the storage means 31. However, this texture image q1 is not transmitted to the edge device 4.
[0107] The pixel reduction processor 33 reduces the number of pixels of the texture image q1 by an appropriate method (for example, downsampling) to generate a reduced pixel number image q2 having n / 4×n / 4 pixels.
[0108] The neural network unit 34 doubles the number of pixels of the low-pixel-count image q2 in both the horizontal and vertical directions to generate a provisional texture image q3 having n / 2×n / 2 pixels.
[0109] The auxiliary information generating unit 35 generates auxiliary information according to the difference between the texture image q1, which is the original image, and the temporary texture image q3. As an example, the auxiliary information generating unit 35 sets the difference image between the two images as the auxiliary information. It is preferable that the auxiliary information generating unit 35 compresses the difference between the two images, and as a specific example, it generates a JPEG image by compressing the difference image, and sets this as the auxiliary information.
[0110] By compressing the difference so that the amount of data required to transmit the auxiliary information is smaller than the amount of data required to transmit the texture image q1, the transmission amount can be reduced compared to transmitting the texture image q1. The auxiliary information generated in this manner is transmitted to the edge device 4 by the transmitter 32.
[0111] On the other hand, the pixel count increasing processor 42 of the edge device 4 has a neural network unit 428 and a texture image generating unit 429 .
[0112] The neural network unit 428 performs the same process as the neural network unit 34 on the (m-2)th texture image r1 having the number of pixels of n / 4×n / 4 transmitted from the edge device 4, and generates the (m-1)th provisional texture image r2 having the number of pixels of n / 2×n / 2. This (m-1)th provisional texture image r2 is the same image as the provisional texture image q3 generated by the cloud server 3.
[0113] The texture image generation unit 429 generates the (m-1)th texture image r3 based on the (m-1)th provisional texture image r2 and auxiliary information from the cloud server 3. As a specific example, the auxiliary information is the above-mentioned JPEG image, and the texture image generation unit 429 adds each pixel in the JPEG image to each pixel in the (m-1)th provisional texture image r2 to generate the (m-1)th texture image r3. As a result, the (m-1)th texture image r3 close to the texture image q1 as the original image held by the cloud server 3 is generated in the edge device 4.
[0114] It should be noted that the compression rate of a JPEG image is variable. Therefore, by lowering the compression rate, it is possible to generate the (m-1)th texture image r3 with high image quality that is closer to the original texture image q1. On the other hand, by increasing the compression rate, it is possible to reduce the amount of data transmitted from the cloud server 3 to the edge device 4. Therefore, the auxiliary information generating unit 35 may set an appropriate compression rate by considering the balance between the image quality of the texture image and the amount of data transmitted.
[0115] In this way, in the third embodiment, a higher quality texture image with a larger number of pixels can be generated while reducing the amount of transmission.
[0116] Any part or all of the functional units described in this specification may be realized by a program. The program mentioned in this specification may be distributed in a non-temporary manner recorded on a computer-readable recording medium, or may be distributed via a communication line (including wireless communication) such as the Internet, or may be distributed in a state in which it is installed on any terminal.
[0117] Based on the above description, a person skilled in the art may be able to conceive additional effects and various modifications of the present invention, but the aspects of the present invention are not limited to the individual embodiments described above. Various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and intent of the present invention derived from the contents defined in the claims and their equivalents.
[0118] For example, what is described in this specification as one device (or component, the same applies below) (including what is depicted in the drawings as one device) may be realized by multiple devices. Conversely, what is described in this specification as multiple devices (including what is depicted in the drawings as multiple devices) may be realized by one device. Alternatively, some or all of the means or functions included in a certain device (e.g., a server) may be included in another device (e.g., a user terminal). Furthermore, a "system" may be composed of one device, or may be composed of two or more devices (e.g., a server and a user terminal, or multiple user terminals).
[0119] Furthermore, not all of the features described in this specification are essential requirements, and in particular, any feature described in this specification but not included in the claims may be considered as optional additional features.
[0120] It should be noted that the applicant is merely aware of the inventions disclosed in the documents in the "Prior Art Documents" column of this specification, and the present invention does not necessarily aim to solve the problems of the inventions disclosed in the documents. The problems that the present invention aims to solve should be identified by taking into consideration the entire specification. For example, if the specification states that a specific effect is achieved by a specific configuration, it can also be said that the problem that is the reverse of the specific effect is solved. However, it is not necessarily intended that such a specific configuration is an essential requirement. [Explanation of symbols]
[0121] 100 Image Processing System 1 Pretreatment section 11 High pixel count processing section 12 Definition improvement section 121 Edge Enhancement Filter 122 Edge-preserving noise reduction filter 2. Neural Network Department 200,201 Texture Mapping System 3. Cloud Server 31 Memory means 32 Transmitter 33 Pixel reduction processing section 34 Neural Network Department 35 Auxiliary information generation section 4. Edge Devices 41 Receiving section 42 High pixel count processing section 421~425,428 Neural Network Division 426 Division 427 Joint 429 Texture Image Generation Unit 43 Texture mapping section 44 Memory means
Claims
1. An image processing device that generates an image to be input to a neural network unit that improves the definition of the image, a pixel count increasing processor that increases the number of pixels of a first image to be processed to generate a second image; a definition enhancing unit that enhances the definition of the second image to generate an image to be input to the neural network unit.
2. The definition improving unit includes: performing edge enhancement on the second image to generate a third image; The image processing apparatus according to claim 1 , further comprising: a processor for generating an image to be input to said neural network unit by removing noise from said third image while preserving edges.
3. The definition improving unit includes: an edge enhancement filter that performs edge enhancement on the second image to generate a third image; 2. The image processing apparatus according to claim 1, further comprising: an edge-preserving noise removal filter that removes noise from the third image while preserving edges to generate an image to be input to the neural network unit.
4. The image processing device according to claim 3 , wherein the edge enhancement filter is a kernel-based filter.
5. The image processing device according to claim 3 , wherein the edge-preserving noise removal filter is a bilateral filter.
6. An image processing device according to any one of claims 1 to 4, and a neural network unit that improves the sense of detail of the image generated by the image processing device.
7. An image processing method for generating an image to be input to a neural network unit that improves the definition of the image, comprising: generating a second image by increasing the number of pixels of a first image to be processed; and increasing the definition of the second image to generate an image to be input to the neural network unit.
8. generating a first image by reducing the number of pixels of the original image; increasing the number of pixels of the first image to generate a second image; enhancing the detail of the second image to generate a third image; and using the original image and the third image, training a neural network section that improves image definition.
9. generating a second image by increasing the number of pixels of a first image to be processed; enhancing the detail of the second image to generate a third image; and inputting the third image into a trained neural network unit that improves image detail to improve image detail.
10. Equipped with a cloud server and an edge device, the cloud server has a transmission unit configured to transmit one or more first texture images, the maximum number of pixels of which is a first number of pixels, to the edge device; The edge device includes: a receiving unit for receiving the one or more first texture images transmitted from the cloud server; a pixel count increasing processor that generates a second texture image having a second pixel count that is greater than the first pixel count, based on at least one of the one or more first texture images; a texture mapping unit that performs texture mapping based on the second texture image.
11. the pixel count increasing processor generates the second texture image using a neural network; The texture mapping system of claim 10 , wherein the transmitter transmits network parameters of the neural network to the edge device.
12. The pixel count increasing processing unit includes: (1) generating a third texture image based on at least one of the one or more first texture images, the third texture image having a third number of pixels greater than the second number of pixels; or (2) generating a third texture image having the third number of pixels based on the second texture image generated based on at least one of the one or more first texture images; A texture mapping system according to claim 10 or 11.
13. The pixel count increasing processing unit includes: Dividing the second texture image generated based on at least one of the one or more first texture images into a plurality of first divided images; increasing the number of pixels of each of the plurality of first divided images to generate a plurality of second divided images; The texture mapping system according to claim 10 or 11, further comprising: combining the second divided images to generate a third texture image.
14. a number of pixels of the one or more first texture images is the first number of pixels; the number of pixels of the second texture image is the second number of pixels; the number of pixels of each of the first divided images is the first number of pixels; the number of pixels of each of the second divided images is the second number of pixels; the pixel count increasing processor includes a neural network unit that increases the pixel count of an image from the first pixel count to the second pixel count, The neural network unit includes: generating the second texture image based on at least one of the one or more first texture images; and The texture mapping system of claim 13 , further comprising: generating each of the second divided images based on each of the first divided images.
15. The texture mapping system according to claim 13 , wherein the pixel count increasing processor performs a smoothing process on a boundary between one of the second divided images and another of the second divided images when combining the plurality of second divided images.
16. the transmitting unit transmits auxiliary information used in generating the second texture image to the edge device; The texture mapping system according to claim 10 , wherein the pixel count increasing processor generates the second texture image based on at least one of the one or more first texture images and the auxiliary information.
17. The pixel count increasing processing unit includes: a first neural network unit that generates a first provisional texture image having the second number of pixels based on at least one of the one or more first texture images; The texture mapping system according to claim 16 , further comprising a texture image generator that generates the second texture image based on the provisional texture image and the auxiliary information.
18. The cloud server includes: a pixel reduction processing unit that reduces the number of pixels of the original image having the second pixel number to generate a reduced pixel number image; a second neural network unit that generates a second provisional texture image based on the low pixel count image by the same process as that of the first neural network unit; 18. The texture mapping system according to claim 17, further comprising: an auxiliary information generating unit configured to generate the auxiliary information according to a difference between the original image and the second temporary texture image.
19. The texture mapping system according to claim 18 , wherein the auxiliary information generator generates the auxiliary information by compressing the difference.
20. 20. The texture mapping system of claim 18, wherein an amount of data required to transmit the auxiliary information to the edge device is less than an amount of data required to transmit the original image to the edge device.
21. a cloud server transmitting one or more first texture images to the edge device, the first texture images having a maximum number of pixels being a first number of pixels; receiving, by an edge device, the one or more first texture images transmitted from the cloud server; generating, by the edge device, a second texture image based on at least one of the one or more first texture images, the second texture image having a second number of pixels greater than the first number of pixels; and performing texture mapping by the edge device based on the second texture image.
22. a receiving unit that receives one or more first texture images, the maximum number of pixels of which is a first number of pixels, transmitted from a cloud server; a pixel count increasing processor that generates a second texture image having a second pixel count that is greater than the first pixel count, based on at least one of the one or more first texture images; A texture mapping unit that performs texture mapping based on the second texture image.
23. receiving, by the edge device, one or more first texture images transmitted from a cloud server, the first texture images having a maximum number of pixels being a first number of pixels; generating, by the edge device, a second texture image based on at least one of the one or more first texture images, the second texture image having a second number of pixels greater than the first number of pixels; and performing texture mapping by the edge device based on the second texture image.
Citation Information
Patent Citations
Learning method, image processing method, and program
JP2021190814A
Image processing method and device, electronic device, and storage medium
JP2021528742A
Information processing apparatus, information processing method, and program
JP2023047600A
Generating image blending weights
JP2023098796A
Machine learning based image adjustment
US20210360179A1
