Object rendering method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses an object rendering method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: shooting a picture of a to-be-rendered object through a shooting device carried by a projector; performing semantic segmentation on the to-be-rendered object in the photo to obtain contour coordinate information of the to-be-rendered object; creating pixel points of a plurality of target colors according to the contour coordinate information of the to-be-rendered object; and projecting the pixel points of the plurality of target colors to the contour position of the to-be-rendered object. According to the scheme, the real object can be rendered.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of projector technology, and specifically to a method, device, electronic device, and computer-readable storage medium for rendering an object. Background Art

[0002] In the field of digital image processing, computer graphics (CG) technology has become a mature technology for rendering halo effects. Rendering digital images based on rendering algorithms, lighting models, and color correction techniques can enhance the artistic expression of images.

[0003] However, compared to creating special effects in digital environments, the exploration of technologies for rendering halo effects for objects in the real world remains a relatively marginal research area. Related technologies use special coating materials to simulate objects, but this method is not as efficient as computer-generated digital image rendering and is not widely applicable. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, electronic device, and computer-readable storage medium for rendering an object, which can render a real object.

[0005] In a first aspect, an embodiment of the present application provides a method for rendering an object, which is applied to a projector; the method comprises:

[0006] Using the camera carried by the projector, take a picture of the object to be rendered;

[0007] Performing semantic segmentation on the object to be rendered in the photo to obtain outline coordinate information of the object to be rendered;

[0008] Creating a plurality of pixel points of a target color according to the outline coordinate information of the object to be rendered;

[0009] Projecting a plurality of pixel points of the target color to the contour position of the object to be rendered.

[0010] In one embodiment, creating a plurality of pixel points of a target color according to the outline coordinate information of the object to be rendered includes:

[0011] Get a solid color image;

[0012] Modifying the color of the pixel points corresponding to the outline coordinate information of the object to be rendered in the solid color image to a target color to obtain a target image; wherein the target color is different from the color of the solid color image;

[0013] The step of projecting a plurality of pixel points of the target color onto the contour position of the object to be rendered includes:

[0014] The target image is projected to the position of the object to be rendered.

[0015] In one embodiment, the step of modifying the color of the pixel points corresponding to the outline coordinate information of the object to be rendered in the solid color image to the target color to obtain the target image includes:

[0016] Converting the outline coordinate information of the object to be rendered into target outline coordinate information in the optical machine coordinate system of the projector;

[0017] The color of the pixel points corresponding to the target outline coordinate information in the solid color image is modified to the target color to obtain the target image.

[0018] In one embodiment, taking a picture of the object to be rendered using a shooting device carried by the projector includes:

[0019] Projecting a white image onto the object to be rendered;

[0020] When projecting the white image, a photograph of the object to be rendered is taken using a photographing device carried by the projector.

[0021] In one embodiment, performing semantic segmentation on the object to be rendered in the photo to obtain outline coordinate information of the object to be rendered includes:

[0022] The semantic segmentation model deployed in the projector is used to perform semantic segmentation on the object to be rendered in the photo to obtain contour coordinate information of the object to be rendered.

[0023] In one embodiment, the semantic segmentation model is a Deeplab-largeFOV model;

[0024] The backbone network of the semantic segmentation model includes a residual module;

[0025] The semantic segmentation model performs batch normalization on the data before performing nonlinear transformation on the data using an activation function.

[0026] In one embodiment, the object to be rendered is an object in an image.

[0027] In a second aspect, an embodiment of the present application provides a device for rendering an object, which is applied to a projector; the device includes:

[0028] A shooting module, configured to take a picture of the object to be rendered using a shooting device carried by the projector;

[0029] A semantic segmentation module, configured to perform semantic segmentation on the object to be rendered in the photo to obtain outline coordinate information of the object to be rendered;

[0030] A creation module, configured to create a plurality of pixel points of a target color according to the outline coordinate information of the object to be rendered;

[0031] The projection module is used to project a plurality of pixel points of the target color to the contour position of the object to be rendered.

[0032] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the above-mentioned object rendering method are implemented.

[0033] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned object rendering method are implemented.

[0034] In a fifth aspect, embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in the embodiments of the present application.

[0035] The embodiments of the present application have the following beneficial effects: a photo of the object to be rendered is taken by a shooting device carried by a projector, thereby converting the real object to be rendered into a digital image, so that the object to be rendered in the photo can be semantically segmented to obtain the contour coordinate information of the object to be rendered; according to the contour coordinate information of the object to be rendered, a plurality of pixel points of the target color can be created, and the pixel points of the target color can be projected by the projector. Because the pixel points of the target color are created according to the contour coordinate information of the object to be rendered, the plurality of pixel points of the target color will be projected at the contour position of the object to be rendered when projected in the direction of the object to be rendered, thereby efficiently achieving a halo effect for rendering the real object to be rendered, thereby enhancing the artistic expression of the object to be rendered. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 This is a schematic diagram of the steps of a method for rendering an object provided in one embodiment of the present application;

[0038] Figure 2 This is an architecture diagram of the DeepLabV1 model provided in one embodiment of the present application;

[0039] Figure 3 Schematic diagram of the principle of dilated convolution provided in one embodiment of the present application;

[0040] Figure 4 This is an architecture diagram of the DeepLab-LargeFOV model provided in one embodiment of the present application;

[0041] Figure 5 is a structural diagram of a residual module provided in one embodiment of the present application;

[0042] Figure 6 This is a flowchart of a method for rendering an object provided by an embodiment of the present application;

[0043] Figure 7 1 is a schematic structural diagram of an object rendering device provided in one embodiment of the present application;

[0044] Figure 8 It is a structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0046] In one embodiment, Figure 1As shown, a method for rendering an object is provided. Although a logical order is shown in the step diagram, in some cases, the steps shown or described may be performed in an order different from that shown in the figure. Specifically, the object rendering method can be applied to a projector, wherein the projector may include but is not limited to one or more of a digital light processing (DLP) projector, a liquid crystal display (LCD) projector, and a liquid crystal on silicon (LCOS) projector.

[0047] It should be noted that the order of description of the following embodiments does not limit the priority order of the embodiments.

[0048] according to Figure 1 The method for rendering an object shown in FIG. 1 includes at least steps S110 to S140, which are described in detail as follows:

[0049] In step S110 , a photograph of the object to be rendered is taken using a photographing device carried by the projector.

[0050] The shooting device carried by the projector can be a device integrated in the projector or a shooting device external to the projector.

[0051] The object to be rendered is a real object with physical properties such as volume and weight. The object to be rendered can be any object such as a mural, a photograph, a sculpture, a plaster statue, or a vase.

[0052] The shooting device carried by the projector can be used to take photos of the object to be rendered, thereby converting the real object to be rendered into a digital image, so that the outline coordinate information of the object to be rendered can be determined based on the photos of the object to be rendered.

[0053] Optionally, the resolution of the photograph of the object to be rendered is the same as the resolution of the projector, so that multiple pixel points of the target color subsequently created according to the contour coordinate information of the object to be rendered can be projected at the original ratio when being projected, avoiding pixel deformation or distortion.

[0054] For example, when the resolution of the projector is 1080P, the resolution of the object to be rendered can also be 1080P.

[0055] In step S120 , semantic segmentation is performed on the object to be rendered in the photo to obtain outline coordinate information of the object to be rendered.

[0056] The projector can have a built-in computing unit that can be used to perform semantic segmentation on the object to be rendered in the photo, identify and classify each pixel in the photo, and thereby determine whether each pixel in the photo is a contour pixel of the object to be rendered. The coordinate information of the contour pixel in the photo is obtained to obtain the contour coordinate information of the object to be rendered.

[0057] Optionally, the computing unit can perform semantic segmentation on the object to be rendered in the photo based on the semantic segmentation technology of deep learning by constructing a model architecture such as a convolutional neural network (CNN) or a transformer model to obtain the contour coordinate information of the object to be rendered.

[0058] Optionally, an encoder can be used to downsample the photo of the object to be rendered to obtain a feature map, thereby extracting high-level semantic features from the photo. The decoder then gradually upsamples the feature map to restore it to the original photo size. Combined with mechanisms such as skip connections, it fuses high- and low-level feature information to enhance segmentation accuracy and detail. The class probability of each pixel is predicted using activation functions such as softmax, and the prediction result is converted into a binary or multi-category mask. Contour detection algorithms (such as Canny edge detection) are then used to extract the contour of the object to be rendered, and the contour coordinate information of the object to be rendered is obtained based on the photo coordinate system.

[0059] Optionally, the object to be rendered may be an object in an image, so as to achieve rendering of the object in the image. When the object to be rendered is an image (such as a mural or a photo), the outline coordinate information of the object to be rendered may be the outline coordinate information of the image, or the outline coordinate information of the object in the image.

[0060] For example, when the object to be rendered is a mural with the content of "horse", the outline coordinate information of the object to be rendered may be the coordinate information of the edge of the mural, or the coordinate information of the edge of the horse in the mural.

[0061] In step S130 , a plurality of pixel points of a target color are created according to the outline coordinate information of the object to be rendered.

[0062] In step S140 , a plurality of pixel points of the target color are projected onto the contour position of the object to be rendered.

[0063] Some projectors can be controlled and programmed in a way that allows them to display only the light corresponding to individual pixels, rather than projecting the entire image. Therefore, multiple pixels of a target color can be created directly based on the coordinates of the object's outline, and then projected onto the object's outline using the projector.

[0064] Most projectors project an entire image rather than individual pixels. Therefore, a pure-color image can be created or acquired. Based on the outline coordinates of the object to be rendered, the color of the pixels corresponding to the outline coordinates of the object to be rendered in the pure-color image is modified to the target color, resulting in a target image. The target image contains multiple pixels of the target color. The pure-color image's color differs from the target color. Given that the pure-color image will be projected onto the object to be rendered, the pure-color image can be a color that has minimal impact on the color of the object to be rendered.

[0065] By projecting the target image onto the location of the object to be rendered, multiple pixels of the target color can be projected onto the outline of the object to be rendered. The projection direction of the projector is the direction of the object to be rendered.

[0066] In this way, by projecting the entire image rather than individual pixels, it can be applied to most projectors without the need for special control and programming of the projector, and the projection of pixels of multiple target colors can be achieved quickly and easily.

[0067] The target color can be set according to actual needs, and the target colors of multiple projected pixels can be consistent or inconsistent.

[0068] For example, when the object to be rendered is a mural with the content of "sun", the target color can be golden yellow, so as to render a halo effect for the sun in the mural.

[0069] For another example, when the object to be rendered is a fruit bowl containing apples and oranges, the target color of the pixels of the apple outline can be red, and the target color of the pixels of the orange outline can be orange, thereby creating a halo effect for the apples and oranges.

[0070] It is understandable that the objects to be rendered may include one or more objects. For example, in the above example, the rendering of apples and oranges may be achieved at the same time.

[0071] By adopting the technical solution of the embodiment of the present application, a photograph of the object to be rendered is taken by a shooting device carried by a projector, thereby converting the real object to be rendered into a digital image, so that the object to be rendered in the photograph can be semantically segmented to obtain the contour coordinate information of the object to be rendered; based on the contour coordinate information of the object to be rendered, multiple pixel points of the target color can be created and projected by the projector. Because the pixel points of the target color are created based on the contour coordinate information of the object to be rendered, the multiple pixel points of the target color will be projected in the contour position of the object to be rendered when projected in the direction of the object to be rendered, thereby efficiently achieving a halo effect for the real object to be rendered, thereby enhancing the artistic expression of the object to be rendered. In addition, the technical solution of the embodiment of the present application can be widely applied to a variety of objects.

[0072] Based on the above technical solution, as an embodiment, the color of the pixel points corresponding to the contour coordinate information of the object to be rendered in the solid color image is modified to the target color to obtain the target image, which may include: converting the contour coordinate information of the object to be rendered into the target contour coordinate information in the coordinate system of the projector's optical machine; and modifying the color of the pixel points corresponding to the target contour coordinate information in the solid color image to the target color to obtain the target image.

[0073] The coordinate information of the outline of the object to be rendered is determined based on a photo of the object to be rendered. The photo of the object to be rendered is taken by a camera carried by the projector. Therefore, the acquired coordinate information of the outline of the object to be rendered is coordinate information in the coordinate system of the camera.

[0074] The projector's optical engine converts image signals into light signals for projection. The target image is projected by the projector's optical engine. The target image is obtained by modifying the color of pixels in a solid-color image. Therefore, the coordinate information in the solid-color image can be determined based on the optical engine's coordinate system.

[0075] In order to ensure that the pixels modified to the target color in the pure color image are the pixels corresponding to the outline coordinate information of the object to be rendered, the coordinate conversion relationship between the coordinate system of the shooting device and the coordinate system of the optical machine can be obtained.

[0076] Based on the coordinate conversion relationship between the coordinate system of the camera and the coordinate system of the optical engine, the outline coordinate information of the object to be rendered is converted into the target outline coordinate information in the coordinate system of the projector's optical engine, thereby modifying the color of the pixel points corresponding to the target outline coordinate information in the solid color image to the target color, thereby obtaining the target image. The calibration method for the coordinate conversion relationship between the coordinate system of the camera and the coordinate system of the optical engine can refer to relevant technologies and is not limited in this application.

[0077] By adopting the technical solution of the embodiment of the present application, the contour coordinate information of the object to be rendered is converted into the target contour coordinate information in the coordinate system of the projector's optical machine, which can ensure that the pixel points modified to the target color in the pure color image are the pixel points corresponding to the contour coordinate information of the object to be rendered, and then determine that when projecting the pure color image, multiple pixel points of the target color can be projected to the contour position of the object to be rendered, thereby realizing the rendering of the object to be rendered.

[0078] Based on the above technical solution, as an embodiment, creating multiple target color pixel points based on the outline coordinate information of the object to be rendered may include: converting the outline coordinate information of the object to be rendered into target outline coordinate information in the coordinate system of the projector's optical engine based on the coordinate conversion relationship between the coordinate system of the camera and the coordinate system of the optical engine, and creating multiple target color pixel points based on the target outline coordinate information. In this way, the multiple target color pixel points can be accurately projected onto the outline positions of the object to be rendered, thereby achieving rendering of the object to be rendered.

[0079] Based on the above technical solution, as an embodiment, taking a photo of the object to be rendered using a shooting device carried by a projector may include: projecting a white image onto the object to be rendered; and while projecting the white image, taking a photo of the object to be rendered using the shooting device carried by the projector.

[0080] In order to avoid poor quality of photos of the object to be rendered in a dimly lit environment, a white image can be projected onto the object to be rendered to achieve the effect of turning on the light.

[0081] When projecting a white image onto the object to be rendered, a photograph of the object to be rendered is taken using a photographing device carried by the projector, so as to improve the quality of the photograph of the object to be rendered.

[0082] By adopting the technical solution of the embodiment of the present application, the quality of the photographs of the object to be rendered can be improved, thereby improving the accuracy of the outline coordinate information of the object to be rendered, so as to achieve a good rendering effect of the object to be rendered.

[0083] Based on the above technical solution, as an embodiment, performing semantic segmentation on the object to be rendered in the photo to obtain the contour coordinate information of the object to be rendered can include: using a semantic segmentation model deployed in a projector to perform semantic segmentation on the object to be rendered in the photo to obtain the contour coordinate information of the object to be rendered.

[0084] The semantic segmentation model can be pre-trained. Considering that the computing resources in the projector are usually less, the semantic segmentation model can be trained on other devices and then deployed in the projector.

[0085] When a photo of the object to be rendered is taken, the semantic segmentation model deployed in the projector is called to perform semantic segmentation processing on the photo of the object to be rendered to obtain the contour coordinate information of the object to be rendered.

[0086] By adopting the technical solution of the embodiment of the present application and utilizing the semantic segmentation model, the semantic segmentation of the objects to be rendered in the photo can be quickly achieved, thereby improving the rendering efficiency of the objects to be rendered.

[0087] Based on the above technical solution, as an embodiment, the semantic segmentation model may be a Deep Lab Large Field Of View (DeepLab-LargeFOV) model.

[0088] Optionally, the backbone network of the semantic segmentation model includes a residual module.

[0089] Optionally, the semantic segmentation model performs batch normalization on the data before performing nonlinear transformation on the data using the activation function.

[0090] The DeepLab-LargeFOV model can be obtained by improving the DeepLab V1 (DeepLabV1) model. The DeepLabV1 model is introduced below.

[0091] Figure 2 This is an architecture diagram of the DeepLabV1 model provided in one embodiment of the present application; the DeepLabV1 model is a convolutional neural network model for semantic segmentation. The DeepLabV1 model combines global context information to better understand the semantic content in the image. Figure 2 ,The DeepLabV1 model includes multiple branches, which extract image features from different scales to capture information,of different scales and improve the perception ability of targets of different sizes.

[0092] The Backbone module of the DeepLabV1 model uses the Visual Geometry Group 16 (VGG16) model architecture as the main convolutional neural network architecture.

[0093] In the DeepLabV1 model, some or all of the fully connected layers of VGG16 are removed, retaining only the convolutional layers. Atrous convolution is used to increase the receptive field, thereby capturing the global context of the image. The VGG16 architecture consists of 16 convolutional and fully connected layers, including 13 convolutional layers and 3 fully connected layers. VGG16 was trained on image datasets and achieved strong performance on image classification tasks.

[0094] In the DeepLabV1 model, the max pooling layer used by VGG16 is slightly different; the convolution kernel of the max pooling layer used by VGG16 in the DeepLabV1 model is 3, the stride is 2, and the padding value is 1. The stride of the last two max pooling layers is set to 1, thereby reducing the downsampling ratio from 32 to 8. The last three 3×3 convolutional layers use dilated convolution with a dilation factor of r=2.

[0095] In the process of convolution of the fully connected layer in the DeepLabV1 model, the first fully connected layer (FC1) downsamples the parameters, and the final result is a convolution layer with a convolution kernel size of 3×3 and a number of convolution kernels of 1024 (expanded convolution). This not only reduces the parameters but also reduces the amount of calculation; for the second fully connected layer (FC2), the number of convolution kernels is also sampled from 4096×4096 to 1024 (ordinary convolution).

[0096] After convolution of FC1, a dilation factor is also set (dilated convolution). For FC2, convolution results in a standard convolution layer with a 1×1 kernel and 1024 kernels. This is followed by another standard convolution layer with a 1×1 kernel and the number of kernels equal to the number of classes (including background). Finally, the image is restored to its original size by upsampling by a factor of 8.

[0097] The DeepLabV1 model retains more spatial information by reducing the use of pooling layers and avoids excessive signal downsampling. The DeepLabV1 model uses upsampling technology to restore the spatial resolution of feature maps, which can better handle pixel-level annotation tasks.

[0098] For semantic segmentation tasks, a large receptive field is very important. The backbone network of the DeepLabV1 model uses dilated convolution to increase the receptive field of the model, reduce model parameters, increase model speed, and have less impact on model performance.

[0099] The Softmax layer in the DeepLabV1 model converts each pixel's category prediction into a probability for the corresponding category. The Softmax layer normalizes the category probability predictions for each pixel, ensuring that each prediction falls between 0 and 1 and that the sum of all category predictions is 1. For each pixel, the probability of the pixel belonging to each category is determined, thereby determining which category has the highest probability of belonging to the pixel.

[0100] The preliminary semantic segmentation results output by the Softmax layer are input into a fully connected conditional random field to further optimize the segmentation boundaries. The fully connected conditional random field combines low-level pixel information (such as color and texture) with high-level semantic information (the prediction results of the neural network) to make the semantic segmentation results more refined and the boundaries more accurate.

[0101] Among them, the dilated convolution, also known as the hole convolution, can expand the receptive field (LargeFOV) and capture multi-scale contextual information. Figure 3 : is a schematic diagram of the principle of dilated convolution provided by an embodiment of the present application, Figure 3 The figure in the figure shows a dilated convolution with a dilation rate of 2. Dilated convolution "expands" the kernel by inserting spaces between its elements, creating a "dilated convolution." The dilation rate parameter, L, indicates the expansion of the kernel, meaning L-1 spaces are inserted between kernel elements. When L = 1, no spaces are inserted between kernel elements, resulting in a standard convolution.

[0102] Multiple sample images can be obtained, and the sample images can be semantically segmented and annotated using a label annotation tool (labelme). The DeepLabV1 model can be trained using the sample images to obtain a trained DeepLabV1 model. The training method can refer to related technologies, such as supervised learning or reinforcement learning.

[0103] Figure 4 This is an architecture diagram of the DeepLab-LargeFOV model provided in one embodiment of the present application. The DeepLab-LargeFOV model used in one embodiment of the present application is based on the DeepLabV1 model architecture, but with multiple branches removed to eliminate multi-scale prediction. This elimination of multi-scale prediction reduces computational overhead, simplifies the model structure, avoids redundant information, improves model robustness, and provides greater scalability, making it easier to expand into practical application scenarios.

[0104] Optionally, a residual module is added to the VGG16 network structure of the DeepLab-LargeFOV model. Figure 5This is a structural diagram of the residual module provided in one embodiment of the present application. By constructing an identity mapping in the residual module, network degradation can be addressed. When the residual is 0, only the identity mapping is performed, and at least the network performance will not degrade. In fact, the residual will not be 0, which enables the stacking layer to learn new features based on the input features, thereby achieving better performance.

[0105] Optionally, the DeepLab-LargeFOV model undergoes model pruning to accelerate model inference. Specifically, the Relu activation function in the DeepLab-LargeFOV model undergoes batch normalization before each. Batch normalization achieves a normalized distribution. By adding L1 regularization as a constraint, the coefficients are sparsely populated, forcing the scaling factor to approach zero. This scaling factor is then used to evaluate channel importance, retaining important channels and removing unimportant ones. In other words, all channels with scaling factors approaching zero are removed.

[0106] As deep networks deepen, the distribution of activation input values ​​gradually shifts, potentially moving toward the upper and lower limits of the range. This results in smaller gradients and slower convergence. Batch Normalization uses certain normalization techniques to normalize the distribution of input values ​​for each neuron layer back to a normal distribution with a mean of 0 and a variance of 1. This increases the gradient, prevents vanishing gradients, accelerates model convergence, improves model generalization, and reduces the model's reliance on parameter initialization.

[0107] After training the DeepLab-LargeFOV model, you can obtain the model .pt file. Use the torch.onnx.export method in PyTorch (an open source deep learning framework for machine learning and deep learning) to convert the .pt file into an onnx file. Because the projector is an Android device, use the onnx2ncnn tool provided by the NCNN framework (a lightweight, high-performance deep learning framework optimized for mobile devices, providing efficient inference performance and low memory usage) to convert the ONNX file into NCNN .bin and .param files (using the command . / onnx2ncnn model.onnxmodel.param model.bin). Finally, use the ncnnoptimize tool to further optimize (using the command . / ncnnoptimize model.param model.bin model_opt.param model_opt.bin 0) to improve inference speed and reduce model size. Finally, deploy the semantic segmentation model on the projector.

[0108] By adopting the technical solution of the embodiment of the present application, the DeepLab-LargeFOV model finally obtained is a relatively lightweight model by canceling multi-scale prediction, adding a residual module and performing model pruning. It can well realize semantic segmentation of images even on a projector with less computing resources.

[0109] Figure 6 It is a flow chart of the object rendering method provided by one embodiment of the present application. In step S1, a sample image set can be obtained in advance; in step S2, the DeepLab-LargeFOV model is trained using the sample image set to obtain a trained DeepLab-LargeFOV model. In step S3, a photo of the object to be rendered is taken by the shooting device carried by the projector. In step S4, the photo is input into the DeepLab-LargeFOV model to obtain the contour coordinate information of the object to be rendered. In step S5, a solid color image is created; in step S6, the color of the pixel points in the solid color image is modified to obtain a target image. In step S7, the target image is projected to the position of the object to be rendered.

[0110] To facilitate better implementation of the object rendering method of the present application, the present application also provides an object rendering device based on the object rendering method. The meanings of the terms herein are the same as those in the object rendering method, and specific implementation details can be referred to the description in the method embodiment.

[0111] See also Figure 7 , Figure 7 : is a schematic structural diagram of an object rendering device provided in an embodiment of the present application, wherein the object rendering device is applied to a projector, and the object rendering device includes:

[0112] The shooting module 701 is used to take a picture of the object to be rendered using a shooting device carried by the projector;

[0113] A semantic segmentation module 702 is configured to perform semantic segmentation on the object to be rendered in the photo to obtain outline coordinate information of the object to be rendered;

[0114] A creation module 703 is used to create a plurality of pixel points of a target color according to the outline coordinate information of the object to be rendered;

[0115] The projection module 704 is configured to project a plurality of pixel points of the target color to the contour position of the object to be rendered.

[0116] In one embodiment, the creation module 703 includes:

[0117] An acquisition unit, used for acquiring a pure color image;

[0118] a modification unit, configured to modify the color of the pixel points corresponding to the outline coordinate information of the object to be rendered in the solid color image to a target color, thereby obtaining a target image; wherein the target color is different from the color of the solid color image;

[0119] The projection module 704 includes:

[0120] A projection unit is used to project the target image to the position of the object to be rendered.

[0121] In one embodiment, the modification unit includes:

[0122] A conversion subunit, configured to convert the outline coordinate information of the object to be rendered into target outline coordinate information in the optical-mechanical coordinate system of the projector;

[0123] The modification subunit is used to modify the color of the pixel points corresponding to the target outline coordinate information in the pure color image to the target color to obtain the target image.

[0124] In one embodiment, the shooting module 701 includes:

[0125] A white projection unit, configured to project a white image onto the object to be rendered;

[0126] When projecting the white image, the shooting unit uses the shooting device carried by the projector to shoot a picture of the object to be rendered.

[0127] In one embodiment, the semantic segmentation module 702 includes:

[0128] The semantic segmentation unit is used to perform semantic segmentation on the object to be rendered in the photo using the semantic segmentation model deployed in the projector to obtain contour coordinate information of the object to be rendered.

[0129] In one embodiment, the semantic segmentation model is a Deeplab-largeFOV model;

[0130] The backbone network of the semantic segmentation model includes a residual module;

[0131] The semantic segmentation model performs batch normalization on the data before performing nonlinear transformation on the data using an activation function.

[0132] In one embodiment, the object to be rendered is an object in an image.

[0133] By adopting the technical solution of the embodiment of the present application, a photo of the object to be rendered is taken by a shooting device carried by a projector, thereby converting the real object to be rendered into a digital image, so that the object to be rendered in the photo can be semantically segmented to obtain the contour coordinate information of the object to be rendered; according to the contour coordinate information of the object to be rendered, a plurality of pixel points of the target color can be created, and the pixel points of the target color can be projected by the projector. Because the pixel points of the target color are created according to the contour coordinate information of the object to be rendered, the plurality of pixel points of the target color will be projected at the contour position of the object to be rendered when projected in the direction of the object to be rendered, thereby efficiently achieving a halo effect for rendering the real object to be rendered, thereby enhancing the artistic expression of the object to be rendered.

[0134] For the specific definition of the object rendering device, please refer to the definition of the object rendering method above and will not be repeated here. Each module in the above-mentioned object rendering device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0135] In addition, the present application also provides an electronic device, such as Figure 8 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0136] The electronic device may include one or more processing core processors 801 and one or more computer readable storage media memories 802 and other components. Those skilled in the art will understand that Figure 8 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0137] The processor 801 is the control center of the electronic device. It connects the various parts of the entire electronic device using various interfaces and lines. By running or executing software programs and / or modules stored in the memory 802 and accessing data stored in the memory 802, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 801 may include one or more processing cores; preferably, the processor 801 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and application programs, and the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 801.

[0138] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 802 may also include a memory controller to provide the processor 801 with access to the memory 802.

[0139] In one embodiment, the electronic device further includes a power supply 803 for supplying power to various components. Preferably, the power supply 803 can be logically connected to the processor 801 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 803 can also include any of one or more DC or AC power supplies, a recharging system, a power supply device debugging circuit, a power converter or inverter, a power status indicator, and other components.

[0140] In one embodiment, the electronic device may further include an input unit 804, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0141] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 801 in the electronic device loads the executable files corresponding to one or more application processes into the memory 802 according to the following instructions, and the processor 801 runs the application stored in the memory 802, thereby implementing the steps of any object rendering method provided in the embodiments of the present application.

[0142] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0143] In one embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in any embodiment of the present application is implemented.

[0144] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in any embodiment of the present application is implemented.

[0145] In some embodiments, a computer program product is also proposed, including a computer program or instructions, which implements the method described in any embodiment of the present application when executed by a processor.

[0146] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0147] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0148] To this end, the present application provides a computer-readable storage medium having a computer program stored thereon. The computer program can be loaded by a processor to execute the steps in any object rendering method provided in the present application.

[0149] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0150] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0151] Since the instructions stored in the computer-readable storage medium can execute the steps in any object rendering method provided in the present application, the beneficial effects that can be achieved by any object rendering method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0152] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements that are inherent to such process, method, article, or terminal device. In the absence of further restrictions, an element defined by the phrase "comprises a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0153] The above is a detailed introduction to the object rendering method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for rendering an object, characterized in that: Applied to a projector; the method comprises: Using the camera carried by the projector, take a picture of the object to be rendered; Performing semantic segmentation on the object to be rendered in the photo to obtain outline coordinate information of the object to be rendered; Creating a plurality of pixel points of a target color according to the outline coordinate information of the object to be rendered; Projecting a plurality of pixel points of the target color to the contour position of the object to be rendered.

2. The method according to claim 1, characterized in that The step of creating a plurality of pixel points of a target color according to the outline coordinate information of the object to be rendered includes: Get a solid color image; Modifying the color of the pixel points corresponding to the outline coordinate information of the object to be rendered in the solid color image to a target color to obtain a target image; wherein the target color is different from the color of the solid color image; The step of projecting a plurality of pixel points of the target color onto the contour position of the object to be rendered includes: The target image is projected to the position of the object to be rendered.

3. The method according to claim 2, characterized in that The step of modifying the color of the pixel points corresponding to the outline coordinate information of the object to be rendered in the solid color image to the target color to obtain the target image includes: Converting the outline coordinate information of the object to be rendered into target outline coordinate information in the optical machine coordinate system of the projector; The color of the pixel points corresponding to the target outline coordinate information in the solid color image is modified to the target color to obtain the target image.

4. The method according to claim 1, wherein The method of taking a photo of the object to be rendered by using a shooting device carried by the projector includes: Projecting a white image onto the object to be rendered; When projecting the white image, a photograph of the object to be rendered is taken using a photographing device carried by the projector.

5. The method according to claim 1, characterized in that The performing semantic segmentation on the object to be rendered in the photo to obtain outline coordinate information of the object to be rendered includes: The semantic segmentation model deployed in the projector is used to perform semantic segmentation on the object to be rendered in the photo to obtain contour coordinate information of the object to be rendered.

6. The method according to claim 5, characterized in that The semantic segmentation model is the Deeplab-largeFOV model; The backbone network of the semantic segmentation model includes a residual module; The semantic segmentation model performs batch normalization on the data before performing nonlinear transformation on the data using an activation function.

7. The method according to any one of claims 1 to 6, characterized in that: The object to be rendered is an object in an image.

8. A device for rendering an object, characterized in that: Applicable to a projector; the device comprises: A shooting module, configured to take a picture of the object to be rendered using a shooting device carried by the projector; A semantic segmentation module, configured to perform semantic segmentation on the object to be rendered in the photo to obtain outline coordinate information of the object to be rendered; A creation module, configured to create a plurality of pixel points of a target color according to the outline coordinate information of the object to be rendered; The projection module is used to project a plurality of pixel points of the target color to the contour position of the object to be rendered.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the object rendering method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the object rendering method according to any one of claims 1 to 7 are implemented.

Images