Deep Learning Methods for Augmented Reality Game Consoles

Abstract

A deep learning method for augmented reality somatosensory game consoles includes the following steps: a processor obtains a depth map based on a depth image and a first mask, wherein the depth image and the first mask are generated based on an image respectively; the processor inputs the depth map and the image into a neural network model to generate a second mask; and the processor generates a background-removed image based on the second mask and the image. The mask obtained by neural network inference in this implementation has a higher accuracy than the original segmentation mask, and the background-removed image of a specific person obtained in the end is more complete, and can still be used simultaneously with heavy-load applications on low-performance platforms.

Description

Technical Field

[0001] This case is about a deep learning method and device for an augmented reality somatosensory game console, and more particularly, about a deep learning method and device for an augmented reality somatosensory game console that uses a deep learning neural network model. Background Art

[0002] Among the deep learning methods for augmented reality somatosensory game consoles, the real-time character background removal technology based on deep learning is a commonly used technology in current social software, and there are also a considerable number of background removal software tools on the market. For example, poor accuracy leads to abnormal damage to the background-removed images, or requires a lot of performance and operation time, and cannot be used for real-time background removal applications on low-performance platforms, and may cause the inability to run other heavy-load applications simultaneously, which greatly reduces availability. Summary of the invention

[0003] One aspect of the present invention is to provide a deep learning method for an augmented reality somatosensory game console, comprising the following steps: a processor obtains a depth map based on a depth image and a first mask, wherein the depth image and the first mask are respectively generated based on an image; the processor inputs the depth map and the image into a neural network model to generate a second mask; and the processor generates a background-removed image based on the second mask and the image.

[0004] In some embodiments, the neural network model further includes an encoding unit and a decoding unit, wherein the processor inputs the depth map and the image into the neural network model to generate the second mask including: the encoding unit generates first data and second data based on the depth map and the image; and the decoding unit synthesizes the first data and the second data to generate a first data set, and decodes the first data set to generate the second mask.

[0005] In some embodiments, the encoding unit generates the first data and the second data based on the depth map and the image, which includes: the MobileNetV2 unit generates the first data based on the depth map and the image; and the MobileNetV2 unit and the ASPP unit generate the second data based on the depth map and the image.

[0006] In some embodiments, the MobileNetV2 unit and the ASPP unit generate the second data based on the depth map and the image, including: performing multiple convolution operations based on multiple convolution parameters and multiple scale parameters to generate multiple first feature data; performing a pooling operation to generate second feature data; combining the first feature data and the second feature data to generate a second data set; and generating the second data based on the second data set.

[0007] In some embodiments, the ratio parameters are mutually prime.

[0008] In some embodiments, synthesizing the first data and the second data by the decoding unit to generate the first data set includes: performing a convolution operation on the first data to generate first feature data; performing a dilation operation on the second data to generate second feature data; and synthesizing the first feature data and the second feature data.

[0009] In some embodiments, decoding the first data set to generate the second mask includes: performing a convolution operation on the first data set to generate third feature data; and performing a dilation operation on the third feature data to generate the second mask.

[0010] In some embodiments, the method further includes: generating the first mask by NUITRACK software; and generating the depth image by a camera.

[0011] Another aspect of the present invention is to provide a deep learning device for an augmented reality somatosensory game console, comprising a processor and a memory. The processor is used to obtain a depth map based on a depth image and a first mask, input the depth map and the image to a neural network model to generate a second mask, and generate a background-removed image based on the second mask and the image, wherein the depth image and the mask are generated based on the image respectively. The memory is used to store the neural network model.

[0012] In some embodiments, the neural network model further includes a MobileNetV2 unit and an ASPP unit, wherein the processor is further used to generate first data based on the MobileNetV2 unit, generate second data based on the MobileNetV2 unit and the ASPP unit, and generate the second mask based on the first data and the second data.

[0013] By means of the above technical scheme, the present invention has at least the following advantages and effects: the mask obtained by the neural network inference of the present invention has higher accuracy than the original segmentation mask, the final background-removed image of the specific person is more complete, and it can still be used simultaneously with heavy-load applications on low-performance platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to make the above and other purposes, features, advantages and embodiments of the present disclosure more clearly understood, the accompanying drawings are described as follows:

[0015] Figure 1 A schematic diagram of a deep learning device for an augmented reality somatosensory game console.

[0016] Figure 2 is a schematic diagram of a deep learning method for an augmented reality somatosensory game console according to some embodiments of the present invention.

[0017] Figure 3 is a schematic diagram of an image captured by a camera according to some embodiments of the present invention.

[0018] Figure 4 is a schematic diagram of a first mask according to some embodiments of the present invention.

[0019] Figure 5 is a schematic diagram of a depth image according to some embodiments of the present invention.

[0020] Figure 6 is a schematic diagram of a depth map according to some embodiments of the present invention.

[0021] Figure 7 is a schematic diagram of a neural network model according to some embodiments of the present invention.

[0022] Figure 8 is a schematic diagram of a second mask according to some embodiments of the present invention.

[0023] Fig. 9 FIG. 4 is a schematic diagram of removing a background image according to some embodiments of the present invention.

[0024]

Main component symbol description

[0025] 100: Deep learning device for augmented reality somatosensory game console 110: Processor

[0026] 130:Memory 150:Camera

[0027] 200: Deep learning method for augmented reality somatosensory game console S210 to S250: Steps 300: Image 400: First Mask

[0028] 500: Depth image 600: Depth map

[0029] 700: Neural network model 710: Encoding unit

[0030] 730: decoding unit 712: MobileNetV2 unit 714:ASPP Unit D1,D2: Data

[0031] 715A to 715D: Convolution operation 718: Pooling operation

[0032] C1 to C5: characteristic data CC1, CC2: data set

[0033] D1C to D3C: Feature data 800: Second mask

[0034] 900: Remove background image DETAILED DESCRIPTION

[0035] The following disclosure provides many different embodiments or examples for implementing different features of the present invention. The components and configurations in the specific examples are used to simplify the present case in the following discussion. Any examples discussed are only used for illustrative purposes and do not limit the scope and significance of the present invention or its examples in any way.

[0036] See also Figure 1 . Figure 1 Schematic diagram of a deep learning device 100 for augmented reality somatosensory game console. Figure 1 As shown, the deep learning device 100 for an augmented reality somatosensory game console includes a processor 110 and a memory 130. The processor 110 is coupled to the memory 130. In some embodiments, the deep learning device 100 for an augmented reality somatosensory game console further includes a camera 150. The camera 150 is coupled to the processor 110.

[0037] See also Figure 2 . Figure 2 FIG. 2 is a schematic diagram of a deep learning method 200 for an augmented reality somatosensory game console according to some embodiments of the present invention. The embodiments of the present invention are not limited thereto.

[0038] It should be noted that the deep learning method 200 of the augmented reality somatosensory game console can be applied to Figure 1 The structure of the deep learning device 100 of the augmented reality somatosensory game console in the embodiment of the present invention is the same or similar system. Figure 2 The operation method is described by taking the example of Figure 1 The application is limited.

[0039] It should be noted that, in some embodiments, the deep learning method 200 for augmented reality somatosensory gaming machine can also be implemented as a computer program and stored in a non-transitory computer-readable medium, so that a computer, an electronic device, or the aforementioned Figure 1 The processor 110 in the deep learning device 100 of the augmented reality somatosensory game console in the embodiment reads the recording medium and performs this operation method. The processor may be composed of one or more chips. The non-transitory computer-readable recording medium may be a read-only memory, a flash memory, a floppy disk, a hard disk, an optical disk, a flash drive, a magnetic tape, a database accessible by a network, or a non-transitory computer-readable recording medium having the same function that can be easily conceived by a person familiar with the art.

[0040] In addition, it should be understood that the operations of the deep learning method of the augmented reality somatosensory game console mentioned in this embodiment can be adjusted in sequence according to actual needs, and can even be executed simultaneously or partially simultaneously, except for those with a special description of the sequence.

[0041] Furthermore, in different embodiments, these operations may be adaptively increased, replaced, and / or omitted.

[0042] See also Figure 2 The deep learning method 200 for augmented reality somatosensory gaming machine includes the following steps.

[0043] In step S210: a depth map is obtained according to the depth image and the first mask. Figure 1 In some embodiments, step S210 may be performed as follows: Figure 1 The detailed operation of step S210 will be referred to in the following. Figures 3 to 6 illustrate.

[0044] In some embodiments, the camera 150 captures an image. The depth image is generated by the NUITRACK software according to the image captured by the camera 150. In some embodiments, the NUITRACK software is stored in the memory 130 and executed by the processor 110.

[0045] See also Figure 3 . Figure 3 is a schematic diagram of an image 300 captured by a camera 150 according to some embodiments of the present invention. It should be noted that the image captured by the camera 150 is a continuous image. Figure 3 The image 300 shown is a frame of a continuous image. In some embodiments, the image 300 is an RGB color image.

[0046] See also Figure 4 . Figure 4 is a schematic diagram of a first mask 400 according to some embodiments of the present invention. Figure 4 The first mask 400 is generated by NUITRACK software based on the image 300. Figure 4 It can be seen that the first mask 400 generated by the NUITRACK software contains defects and leaks, and the effect of directly using it to remove the background of a character is not good.

[0047] See also Figure 5 . Figure 5 is a schematic diagram of a depth image 500 according to some embodiments of the present invention. Figure 1 In some embodiments, the camera 150 is a depth camera. The depth image 500 is one frame of the depth image acquired by the camera 150. Figure 3 Corresponding to the image 300 in FIG.

[0048] See also Figure 6 . Figure 6 is a schematic diagram of a depth map 600 according to some embodiments of the present invention. The depth map 600 is composed of Figure 1 The processor 110 in Figure 5 The depth image 500 and Figure 4 In some embodiments, Figure 1 Processor 110 in the synthesis Figure 5 The depth image 500 and Figure 4 The first mask 400 in the image is used to generate a depth map 600. Figure 4 The first mask 400 is processed Figure 5 After the depth image 500 is obtained, only the depth data of the target person is left in the depth image 500 to generate a depth map 600.

[0049] Please refer back to Figure 2 In step S230, the depth map and the image are input to the neural network model to generate a second mask. Figure 1 In some embodiments, step S230 may be performed as follows: Figure 1 The detailed operation of step S230 will be referred to in the following. Figures 7 and 8 illustrate.

[0050] See also Figure 7 . Figure 7 is a schematic diagram of a neural network model 700 according to some embodiments of the present invention. In step S230, Figure 1 The processor 110 in Figure 3 Images 300 and Figure 6 The depth map 600 in is input to the neural network model 700 to generate a second mask.

[0051] like Figure 7 As shown, the neural network model 700 includes an encoding unit 710 and a decoding unit 730. In some embodiments, in step S230, the encoding unit 710 generates first data D1 and second data D2 according to the depth map 600 and the image 300. In some embodiments, the neural network model 700 is DeepLabv3+.

[0052] In detail, in some embodiments, the encoding unit 710 includes a MobileNetV2 unit 712 and an ASPP (Atrous Spatial Pyramid Pooling) unit 714. The MobileNetV2 unit 712 performs a feature compression operation on the depth map 600 and the image 300 to generate the first data D1. On the other hand, after the MobileNetV2 unit 712 performs a feature compression operation on the depth map 600 and the image 300, the ASPP unit 714 performs a feature extraction operation on the depth map 600 and the image 300 to generate the second data D2.

[0053] like Figure 7 As shown, after the MobileNetV2 unit 712 performs feature compression operations on the depth map 600 and the image 300, the ASPP unit 714 performs multiple convolution operations 715A to 715D on the depth map 600 and the image 300 to generate a plurality of first feature data C1 to C4. In addition, the ASPP unit 714 performs a pooling operation 718 on the depth map 600 and the image 300 to generate a second feature data C5. Then, the ASPP unit 714 synthesizes the plurality of first feature data C1 to C4 and the second feature data C5 to generate a data set CC1.

[0054] In the ASPP unit 714, the convolution operations 715A to 715D respectively include convolution parameters and scaling parameters. The scaling parameters between the convolution operations 715A to 715D are mutually prime.

[0055] For example, in some embodiments, the convolution parameters of the convolution operation 715A are 1×1×256. The convolution parameters of the convolution operation 715B are 3×3×256, and the scale parameter is 1. The convolution parameters of the convolution operation 715C are 3×3×256, and the scale parameter is 2. The convolution parameters of the convolution operation 715D are 3×3×256, and the scale parameter is 3. The scale parameters of the convolution operations 715B to 715D are mutually prime.

[0056] The ASPP unit uses dilated convolution to increase the neural network's receptive field around each pixel, so that it will not only recognize linear features, which can greatly improve the results of the neural network inference and make the resulting mask more complete. The characteristic of dilated convolution is that it will extract features based on the scale parameter (rate) in a grid-like manner, and extract features of a larger field of view with a small number of convolutions. If the scale parameters between different convolution operations share a common factor of more than 1, it will cause the grid effect problem, so that the features of some pixels are not extracted. Therefore, if the scale parameters between different convolution operations are mutually prime, the grid effect problem can be improved.

[0057] Next, the encoding unit 710 performs a 1×1 convolution operation on the data set CC1 to generate second data D2.

[0058] After receiving the first data D1 and the second data D2 from the encoding unit 710 , the decoding unit 730 synthesizes the first data D1 and the second data D2 to generate a data set CC2 . The decoding unit 730 also generates a second mask according to the data set CC2 .

[0059] See also Figure 8 . Figure 8 is a schematic diagram of a second mask 800 according to some embodiments of the present invention. Figure 4 Compared with the first mask 400 in FIG. 4 , the second mask 800 has a higher accuracy.

[0060] Specifically, in some embodiments, the decoding unit 730 performs a 1×1 convolution operation on the first data D1 to generate the first feature data D1C. The decoding unit 730 performs a dilation operation with a parameter of 4 on the second data D2 to generate the second feature data D2C. The decoding unit 730 synthesizes the first feature data D1C and the second feature data D2C to generate the data set CC2. The decoding unit 730 then performs a 3×3 convolution operation on the data set CC2 to generate the third feature data D3C, and then performs a dilation operation on the third feature data D3C to generate the second mask.

[0061] In some embodiments, the neural network model 700 includes a loss function. The loss function is expressed as follows:

[0062]

[0063] in is the inferred mask pixel transparency, is the pixel transparency of the correct answer, and ∈ is a constant to prevent the neural network from having no gradient to update when the transparency is all 0.

[0064] Please refer back to Figure 2 In step S250, a background-removed image is generated according to the second mask and the image. Figure 1 In some embodiments, step S250 may be performed as follows: Figure 1 Executed by the processor 110 in Fig. 9 . Fig. 9 FIG. 9 is a schematic diagram of a background-removed image 900 according to some embodiments of the present invention. In some embodiments, the background-removed image 900 is a Figure 8 The second mask 800 shown in FIG. Figure 3 The image 300 shown in is synthesized.

[0065] In some embodiments, the processor 250 may be a server or other device. In some embodiments, the processor 110 may be a server, circuit, central processor unit (CPU), microprocessor (MCU) or other device with equivalent functions having functions such as storage, calculation, data reading, receiving signals or information, and transmitting signals or information. In some embodiments, the camera 150 may be a circuit or other device with equivalent functions having functions such as image capture and photo taking. In some embodiments, the memory 130 may be a component with storage function or a component with similar functions.

[0066] There are relatively mature products for character detection and segmentation on the market, such as NuiTrack SDK. The masks obtained by segmentation often have various defects and leaks, and the effect of directly using them for character back removal is not good. From the above-mentioned implementation methods, it can be seen that the embodiments of this case provide a deep learning method for an augmented reality somatosensory game console and an augmented reality device thereof, and use a neural network to refer to the information of the broken masks generated by products such as NuiTrack SDK to infer the complete mask, which can be used to produce the correct back-removed image of a specific person. In addition, since the network structure of DeepLabv3+ has a very large amount of computation, each inference takes a long time. Therefore, replacing the neural network for preliminary feature extraction from Xception_101 to MobileNetV2 can greatly reduce the time consumption of this stage, improve the speed of inference, and realize real-time application of dynamic image back removal.

[0067] In summary, the mask obtained by neural network inference in the implementation method of this case has higher accuracy than the original segmentation mask, and the final image of the specific person with the background removed is more complete, and can still be used simultaneously with heavy-load applications on low-performance platforms.

[0068] In addition, the above examples include sequential exemplary steps, but the steps do not have to be performed in the order shown. Performing the steps in different orders is within the scope of the present disclosure. Within the spirit and scope of the embodiments of the present disclosure, the steps may be added, replaced, changed in order, and / or omitted as appropriate.

[0069] Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope of the patent application.

Claims

1. A deep learning method for augmented reality somatosensory game console, characterized in that: Include: A processor obtains a depth map according to the depth image and the first mask, wherein the depth image and the first mask are respectively generated according to the image; Inputting the depth map and the image into a neural network model by the processor to generate a second mask; and The processor generates a background-removed image according to the second mask and the image; The neural network model further comprises an encoding unit and a decoding unit, wherein the processor inputs the depth map and the image into the neural network model to generate the second mask comprising: The encoding unit generates first data and second data according to the depth map and the image; and The decoding unit synthesizes the first data and the second data to generate a first data set, and decodes the first data set to generate the second mask; The first data and the second data generated by the encoding unit according to the depth map and the image include: The MobileNetV2 unit generates the first data according to the depth map and the image; and The MobileNetV2 unit and the ASPP unit generate the second data according to the depth map and the image.

2. The deep learning method for augmented reality somatosensory game console according to claim 1, characterized in that: The second data generated by the MobileNetV2 unit and the ASPP unit according to the depth map and the image includes: Performing a plurality of convolution operations according to a plurality of convolution parameters and a plurality of scale parameters to generate a plurality of first feature data; Performing a pooling operation to generate second feature data; Combining the first feature data and the second feature data to generate a second data set; and The second data is generated according to the second data set.

3. The deep learning method for augmented reality somatosensory game console according to claim 2, characterized in that: The scale parameters are mutually prime.

4. The deep learning method for augmented reality somatosensory game console according to claim 1, characterized in that: The decoding unit synthesizing the first data and the second data to generate the first data set includes: Performing a convolution operation on the first data to generate first feature data; Performing an expansion operation on the second data to generate second feature data; and The first feature data and the second feature data are synthesized.

5. The deep learning method for augmented reality somatosensory game console according to claim 1, characterized in that: Wherein decoding the first data set to generate the second mask comprises: Performing a convolution operation on the first data set to generate third feature data; and The third feature data is dilated to generate the second mask.

6. The deep learning method for augmented reality somatosensory game console according to claim 1, characterized in that: Also includes: The first mask is generated by NUITRACK software; and The depth image is generated by a camera.

7. A deep learning device for an augmented reality somatosensory game console, characterized in that: Include: A processor, configured to obtain a depth map according to the depth image and the first mask, input the depth map and the image into a neural network model to generate a second mask, and generate a background-removed image according to the second mask and the image, wherein the depth image and the mask are respectively generated according to the image; as well as Memory, used to store the neural network model; The neural network model further includes a MobileNetV2 unit and an ASPP unit, wherein the processor is further used to generate first data based on the MobileNetV2 unit, generate second data based on the MobileNetV2 unit and the ASPP unit, and generate the second mask based on the first data and the second data.

Images