Projection control method and device, electronic equipment and storage medium

Abstract

The invention discloses a projection control method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the three-dimensional reconstruction of a real projection region, and obtaining a virtual projection region; shooting the virtual projection area through a first virtual camera located at the observation visual angle to obtain a virtual shot image under the corresponding observation visual angle; adjusting the texture of a target area in the virtual shot image under the observation visual angle to obtain a target image under the observation visual angle; projecting a target image under the observation visual angle to a virtual projection area through a virtual projector located at the observation visual angle; shooting the virtual projection area where the target image is projected through a second virtual camera to obtain a target shot image; and the target shot image is projected in the real projection area through the real projector. In the application, the target image can accurately reflect the creation intention of the user in real time, the immersion degree of user experience and the creation flexibility are improved, and then the real-time projection effect is improved.

Description

Technical Field

[0001] The present application relates to the field of image projection technology, and more specifically, to a projection control method, device, electronic device and storage medium. Background Art

[0002] Currently, virtual reality technology faces challenges in content creation and interaction, which are mainly reflected in the inefficiency of the creation process and the non-real-time nature of user interaction.

[0003] Traditional methods rely on users to manually design and draw the projection content that the projector needs to project to realize virtual reality technology. Manual design and drawing of projection content is time-consuming, which makes it difficult for the projected content to reflect the user's creative intent in real time and accurately, limiting the immersion of the user experience and the flexibility of creation, resulting in poor real-time projection effects. Summary of the invention

[0004] In view of the above problems, the present application proposes a projection control method, device, electronic device and storage medium, which can improve projection diversity.

[0005] In a first aspect, an embodiment of the present application provides a projection control method, the method comprising:

[0006] Perform three-dimensional reconstruction on the real projection area to obtain a virtual projection area;

[0007] The virtual projection area is photographed by a first virtual camera located at an observation angle to obtain a virtual photographed image at a corresponding observation angle;

[0008] Adjusting the texture of the target area in the virtual captured image under the observation angle to obtain the target image under the observation angle;

[0009] The target image in the observation angle is projected into the virtual projection area by a virtual projector located in the observation angle;

[0010] The virtual projection area on which the target image is projected is photographed by a second virtual camera to obtain a target photographed image; the position of the second virtual camera is determined based on the position of the real projector;

[0011] The target image is captured by a real projector in the real projection area.

[0012] In a second aspect, an embodiment of the present application provides a projection control device, the device comprising:

[0013] A reconstruction module is used to perform three-dimensional reconstruction of the real projection area to obtain a virtual projection area;

[0014] A first shooting module is used to shoot the virtual projection area through a first virtual camera located at an observation angle to obtain a virtual shot image at a corresponding observation angle;

[0015] A texture adjustment module, used to adjust the texture of the target area in the virtual captured image under the observation angle to obtain the target image under the observation angle;

[0016] A projection module, used for projecting a target image under the observation angle into a virtual projection area through a virtual projector located at the observation angle;

[0017] A second shooting module is used to shoot the virtual projection area where the target image is projected through a second virtual camera to obtain a target shot image; the position of the second virtual camera is determined based on the position of the real projector;

[0018] The control module is used to project the target captured image into the real projection area through the real projector.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory; the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above method is implemented.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the above-mentioned method is implemented.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product or a computer program including computer instructions, which implement the above method when executed by a processor.

[0022] The embodiments of the present application provide a projection control method, device, electronic device and storage medium. In the present application, a real projection area is three-dimensionally reconstructed to obtain a virtual projection area, and then the virtual projection area is photographed by a virtual camera of an observation angle to obtain a virtual photographed image under the observation angle, and then the texture of the target area in the virtual photographed image under the observation angle is adjusted to obtain the target image under the observation angle, and then the target image is projected by a virtual projector under the observation angle, and the virtual projection area on which the target image is projected is photographed by a second virtual camera to obtain the target photographed image, and finally the real projector projects the target photographed image in the real projection area so that the target area can present the texture in the target image, thereby realizing real-time adjustment of the texture of the target area in the real projection area, and does not require manual design and drawing by the user to realize the texture adjustment of the target area, and the acquisition efficiency of the target image is high, so that the target image can reflect the user's creative intention in real time and accurately, thereby improving the immersion of the user experience and the flexibility of creation, thereby improving the real-time projection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A flowchart of a projection control method proposed in one embodiment of the present application is shown.

[0025] Figure 2 A schematic diagram of a stereo matching process in an embodiment of the present application is shown.

[0026] Figure 3 A schematic diagram of a triangulation method in an embodiment of the present application is shown.

[0027] Figure 4 A schematic diagram of a virtual captured image in an embodiment of the present application is shown.

[0028] Figure 5 Shows Figure 4 A schematic diagram of a target image corresponding to the captured image in .

[0029] Figure 6 Shows Figure 4 Schematic diagram of another target image corresponding to the captured image in .

[0030] Figure 7 Shows Figure 4 Schematic diagram of another target image corresponding to the captured image in .

[0031] Figure 8 A schematic diagram of observing a target object at a viewing angle in an embodiment of the present application is shown.

[0032] Fig. 9 A schematic diagram of observing a target object at another viewing angle in an embodiment of the present application is shown.

[0033] Fig.10 A schematic diagram of observing a target object at another viewing angle in an embodiment of the present application is shown.

[0034] Fig.11 A block diagram of a projection control device according to an embodiment of the present application is shown.

[0035] Fig.12 A block diagram of an electronic device according to an embodiment of the present application is shown.

[0036] Fig.13A structural block diagram of a computer-readable storage medium according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0040] Reference Figure 1 , Figure 1 A flowchart of a projection control method according to an embodiment of the present application is shown. The method is used in an electronic device and includes:

[0041] S110, performing three-dimensional reconstruction on the real projection area to obtain a virtual projection area.

[0042] In the present application, the electronic device may be an independent device or a real projector, and the real projector implements part or all of the functions of the projection control method in the present application.

[0043] In this embodiment, the real projection area refers to the real area where the real projector projects the projection image. The real projection area can be reconstructed in three dimensions to obtain a three-dimensional virtual area as the virtual projection area corresponding to the real projection area.

[0044] As an implementation manner, S110 may include: acquiring a real shot image of the real projection area through a real camera; and performing three-dimensional reconstruction of the real projection area according to the number of real cameras and the real shot image to obtain a virtual projection area.

[0045] The target projection image may be any target projection image, such as a person image, an animal image, a building image, a landscape image, etc. The target projection image may also be a structured light image. The structured light image refers to a specific image with coded information, such as a phase coded image, a Gray code image, a pseudo-random array image, etc.

[0046] First, the target real projector and the real camera can be fixed and their positions can be kept fixed. The target projection image is projected to the real projection area through the target real projector, and then the real projection area is photographed by the real camera to obtain the real photographed image. Afterwards, the real projection area is three-dimensionally reconstructed according to the number of real cameras and the real photographed image to obtain the virtual projection area. Among them, the target real projector can be a real projector under any observation angle, and the target location of the target real projector is used to construct the virtual projection area. Therefore, this application does not limit the setting position of the target real projector.

[0047] Specifically, according to the number of real cameras and the real captured images, the real projection area is three-dimensionally reconstructed to obtain the virtual projection area, including: determining the point cloud corresponding to the real projection area according to the number of real cameras and the real captured images; performing at least one of outlier removal processing, filtering processing and hole filling processing on the point cloud to obtain an intermediate point cloud; performing triangle facet filtering processing on multiple triangular facets generated according to the intermediate point cloud to obtain the virtual projection area.

[0048] Among them, in some embodiments, the point cloud corresponding to the real projection area is determined according to the number of real cameras and the real captured images, including: if the number of real cameras is one, the point cloud corresponding to the real projection area is determined according to the relative relationship between the real camera and the target real projector, the real captured image and the target projection image.

[0049] First, the target real projector and the real camera can be fixed, and then the target real projector can be controlled to project a structured light image to the real projection area, and then the real camera can shoot the projection area to obtain a real shot structured light image. After that, the structured light in the real shot structured light image can be decoded to obtain the relative relationship between the target real projector and the real camera. The relative relationship between the target real projector and the real camera is also called the external parameter between the target real projector and the real camera (including the translation and rotation between the target real projector and the real camera). Through the relative relationship between the target real projector and the real camera, the corresponding relationship between the projection pixels in the projection image projected by the target real projector and the shooting pixels in the shooting image shot by the real camera can be obtained (for example, through the relative relationship between the target real projector and the real camera, it can be determined that the shooting pixel corresponding to the projection pixel a1 in the projection image is b1).

[0050] After obtaining the relative relationship between the target real projector and the real camera, the point cloud corresponding to the real projection area is determined by using the triangulation method directly according to the relative relationship between the target real projector and the real camera, the real captured image and the target projection image.

[0051] In other embodiments, the point cloud corresponding to the real projection area is determined according to the number of real cameras and the real captured images, including: if the number of real cameras is greater than one, the point cloud corresponding to the real projection area is determined according to the relative relationship between the real cameras and the real captured images of each real camera.

[0052] If there are at least two real cameras, each real camera can be fixed and their positions can be kept unchanged. When the target real projector projects the structured light image to the real projection area, the real structured light image is obtained through each real camera. According to the real structured light images obtained by each real camera, the relative relationship between each real camera is determined with the decoding value corresponding to the structured light image as the medium. The relative relationship between each real camera is also called the external parameter between each real camera (including the translation and rotation between each real camera).

[0053] Among them, epipolar correction is performed on the real-shot structured light images taken by each real camera, so that the epipolar lines of the real-shot structured light images taken by each real camera are horizontal and the pixel rows are aligned, thereby realizing the correction of the real-shot structured light images taken by each real camera; then, the relative relationship between each real camera can be determined by global matching according to the corrected images corresponding to the real-shot structured light images taken by each real camera, or the relative relationship between each real camera can be determined by stereo matching.

[0054] Among them, stereo matching means: matching the pixels in the same row of the rectified image corresponding to the real structured light image taken by each real camera, obtaining the correspondence between the pixels in the rectified image corresponding to the real structured light image taken by each real camera, and finally determining the relative relationship between each real camera according to the correspondence between the pixels in the rectified image corresponding to the real structured light image taken by each real camera.

[0055] When there are two real cameras, Figure 2As shown, after determining the corrected image corresponding to the real structured light image taken by the real camera, any one of the corrected images is selected as the fixed image 201, and another corrected image is selected as the image to be searched 202. For a pixel 20111 in any pixel row 2011 in the fixed image 201, the pixel row 2021 matching the pixel row 2011 is directly determined in the image to be searched 202, and then the pixel matching the pixel 20111 is searched in the pixel row 2021. It is not necessary to search for the pixel matching the pixel 20111 in the entire image to be searched 202, thereby greatly reducing the matching time and improving the efficiency of determining the relative relationship between each real camera.

[0056] After the relative relationship between the real cameras is obtained, the point cloud corresponding to the real projection area can be determined by using the triangulation method according to the relative relationship between the real cameras and the real captured image of each real camera.

[0057] like Figure 3 As shown, when there are three real cameras, the pixel point x1 in the captured image 301 corresponding to the real camera 1, the pixel point x2 in the captured image 302 corresponding to the real camera 2, and the pixel point x3 in the captured image 301 corresponding to the real camera 3 are pixels corresponding to a real point in the real projection area. A straight line s1 is constructed by the optical center o1 of the real camera 1 and the pixel point x1 in the captured image 301, a straight line s2 is constructed by the optical center o2 of the real camera 2 and the pixel point x2 in the captured image 302, and a straight line s3 is constructed by the optical center o3 of the real camera 3 and the pixel point x3 in the captured image 303. The point x with the same distance to the three straight lines s1, s2 and s3 is determined to be the real point corresponding to x1, x2 and x3.

[0058] After constructing the point cloud corresponding to the real projection area, the point cloud can be further processed by at least one of outlier removal, filtering and hole filling to obtain an intermediate point cloud. Among them, outlier removal refers to removing isolated points in the point cloud, filtering can include mean filtering, median filtering, etc.; hole filling can include mean hole filling, median hole filling, etc.

[0059] After obtaining the intermediate point cloud, a triangular patch can be directly constructed based on the intermediate point cloud; if the number of point clouds in the intermediate point cloud exceeds the set number threshold, downsampling is performed to obtain a sampled point cloud; downsampling includes random downsampling, voxel downsampling, etc.; then, triangular patches are continuously constructed based on the sampled point cloud. Among them, the method of constructing triangular patches can include the Poisson method and the rolling ball method.

[0060] Then, the constructed triangles can be subjected to triangle filtering processing to filter out inaccurate triangles to obtain filtered triangles. The triangle filtering processing can include: setting a distance threshold of the point cloud, calculating the minimum distance between the vertex coordinates of the triangle and the point cloud, filtering out triangles whose minimum distance is greater than the distance threshold, and obtaining filtered triangles; the triangle filtering processing can include: setting a depth threshold, calculating the difference between the vertex depth of the triangle and the corresponding point cloud depth, filtering out triangles whose depth difference is greater than the depth threshold, and obtaining filtered triangles.

[0061] After obtaining the filtered triangles, the normal vector of each triangle can be further calculated, and UV (U in the horizontal direction and V in the vertical direction) mapping can be performed according to the normal vector of each triangle to obtain a virtual projection area.

[0062] After obtaining the virtual projection area, the virtual projection area can be imported into the virtual system, and then a virtual environment can be constructed in the virtual system through the unity tool, where the virtual environment at least includes the virtual projection area.

[0063] S120: photographing the virtual projection area by using a first virtual camera located at an observation angle to obtain a virtual photographed image corresponding to the observation angle.

[0064] The observation angle may include an observation direction and an observation distance (the observation distance may refer to the distance between the first virtual camera and the target object, and the observation direction may refer to the direction pointed by the optical axis of the first virtual camera). At least one observation angle may be set in the real projection area based on the requirements. For example, four observation angles may be set: an observation angle in front, an observation angle in the back, an observation angle in the left, and an observation angle in the right.

[0065] If there is only one observation angle, a virtual camera can be placed at a position corresponding to the observation angle in the virtual projection area. The virtual camera is the first virtual camera under the observation angle, and the first virtual camera is allowed to shoot in the observation direction of the observation angle to obtain a virtual captured image under the observation angle. At this time, the obtained virtual captured image is one.

[0066] If there are multiple observation viewing angles and there is one first virtual camera, then for each observation viewing angle, the first virtual camera is set at a position corresponding to the observation viewing angle in the virtual projection area, and the virtual projection area is photographed by the first virtual camera to obtain a virtual photographed image under the observation viewing angle, and each observation viewing angle is traversed in this way to obtain a virtual photographed image under each observation viewing angle.

[0067] If there are multiple observation viewing angles and multiple first virtual cameras, a virtual camera is set at a position corresponding to each observation viewing angle in the virtual projection area as a first virtual camera. The virtual projection area is photographed by the first virtual camera located at each observation viewing angle to obtain a virtual photographed image at each observation viewing angle.

[0068] S130: Adjust the texture of the target area in the virtual captured image under the observation viewing angle to obtain the target image under the observation viewing angle.

[0069] The target area may be an area selected in the virtual projection area. In the present application, the target real area may be determined in the real projection area, and then the area corresponding to the target real projection area in the virtual projection area is obtained as the target area, wherein any area in the real projection area may be selected as the target real area, or an area of ​​a specific shape in the real projection area may be designated as the target real projection area, and the specific shape may be, for example, convex, concave, or uneven.

[0070] For each observation viewing angle, the texture of the target area in the virtual captured image under the observation viewing angle is adjusted to the required texture, and the texture of the area other than the target area in the virtual captured image under the observation viewing angle is kept unchanged, and the adjusted virtual captured image under the observation viewing angle is obtained as the target image under the observation viewing angle.

[0071] As an implementation method, S130 may include: obtaining an image mask corresponding to the target area under the observation angle; the image mask corresponding to the virtual captured image is used to indicate the area where the target area is located in the virtual captured image; based on the virtual captured image under the observation angle and the image mask corresponding to the virtual captured image under the observation angle, adjusting the texture of the target area in the virtual captured image under the observation angle to obtain the target image under the observation angle.

[0072] For each observation angle, the virtual captured image under the observation angle can be segmented to obtain a first area indicating the target area and a second area excluding the first area, and then the pixel values ​​of the pixels in the first area are adjusted to the first target value (e.g., 1), and the pixel values ​​of the pixels in the second area are adjusted to the second target value (e.g., 0), to obtain an image mask corresponding to the virtual captured image under the observation angle, so that the image mask corresponding to the virtual captured image is used to indicate the area where the target area is located in the virtual captured image. The virtual captured image can be segmented by manual segmentation or by a segmentation model; the segmentation model can be constructed by a convolutional neural network, a long short-term memory neural network, etc.

[0073] For each observation viewing angle, after determining the image mask corresponding to the virtual captured image under the observation viewing angle, the texture of the target area in the virtual captured image under the observation viewing angle can be adjusted according to the virtual captured image under the observation viewing angle and the corresponding image mask to obtain the target image under the observation viewing angle.

[0074] As an implementation method, for each observation perspective, texture description information for the target object can be obtained; through an artificial intelligence content generation model, based on the texture description information and the image mask corresponding to the virtual captured image under the observation perspective, the texture of the pixel area where the target area is located in the virtual captured image under the observation perspective is adjusted to the texture indicated by the texture description information, so as to obtain the target image under the observation perspective.

[0075] The artificial intelligence content generation model (AIGC) may refer to a model capable of generating images, and may be a model constructed by a convolutional neural network, a long short-term memory neural network, etc. Texture description information refers to information used to describe texture, for example, the texture description information is red and blue stripes, pure red, or a cartoon pattern, etc.

[0076] For each observation viewing angle, the texture description information and the image mask corresponding to the virtual captured image at the observation viewing angle can be directly input into the artificial intelligence content generation model. The artificial intelligence content generation model is used to adjust the texture of the pixel area where the target area in the virtual captured image at the viewing angle is located to the texture indicated by the texture description information, under the constraint of the image mask corresponding to the virtual captured image at the viewing angle, so as to generate the target image at the observation viewing angle.

[0077] For each observation viewing angle, the target image under the observation viewing angle can be determined in the above-mentioned manner.

[0078] S140: Projecting the target image under the observation viewing angle into the virtual projection area through a virtual projector located at the observation viewing angle.

[0079] If there is one observation viewing angle, a virtual projector is set at a position corresponding to the observation viewing angle in the virtual projection area, and then the target image under the observation viewing angle is projected to the virtual projection area through the virtual projector under the observation viewing angle.

[0080] If there are multiple observation viewing angles, for each observation viewing angle, a virtual projector is set at a position corresponding to the observation viewing angle in the virtual projection area, and then the target image under the observation viewing angle is projected to the virtual projection area by the virtual projector under the observation viewing angle. In this way, multiple virtual projectors are controlled to project the corresponding target images to the virtual projection area respectively, so that the target shooting images of multiple observation viewing angles cover the virtual projection area.

[0081] S150 , photographing the virtual projection area on which the target image is projected by a second virtual camera to obtain a target photographed image.

[0082] Among them, the position of the second virtual camera is determined based on the position of the real projector; the real projection position of the real projector in the real projection area can be determined, and then the virtual projection position corresponding to the real projection position is determined in the virtual environment, and a virtual camera is set at the virtual projection position. The virtual camera serves as the second virtual camera, and the shooting angle of the second virtual camera is consistent with the real projection angle of the real projector. In this way, the image captured by the second virtual camera can be directly projected by the real projector.

[0083] After the virtual projectors located at each of the observation viewing angles project the target images at each of the observation viewing angles onto the virtual projection area, the virtual projection area onto which the target images are projected is photographed by a second virtual camera, and the obtained photographed images are used as the target photographed images.

[0084] In a specific embodiment, there may be only one real projector, and in this case there is also only one second virtual camera.

[0085] In another specific embodiment, there may be multiple real projectors, and accordingly, multiple second virtual cameras corresponding to the real projectors are provided.

[0086] S160: Use the real projector to project the target captured image into the real projection area.

[0087] After obtaining the target captured image, the target captured image can be projected into the real projection area through the real projector. The target real area corresponding to the target area in the real projection area can present the texture corresponding to the target area in the target image, thereby realizing texture adjustment of the target real area.

[0088] In this embodiment, a three-dimensional reconstruction is performed on the real projection area to obtain a virtual projection area, and then the virtual projection area is photographed by a virtual camera of the observation angle to obtain a virtual photographed image under the observation angle, and then the texture of the target area in the virtual photographed image under the observation angle is adjusted to obtain the target image under the observation angle, and then the target image is projected by a virtual projector under the observation angle, and the virtual projection area on which the target image is projected is photographed by a second virtual camera to obtain a target photographed image, and finally the real projector projects the target photographed image into the real projection area, so that the target area can present the texture in the target image, and real-time adjustment of the texture of the target area in the real projection area is realized, and there is no need for the user to manually design and draw to realize the texture adjustment of the target area, and the acquisition efficiency of the target image is high, so that the target image can reflect the user's creative intention in real time and accurately, thereby improving the immersion of the user experience and the flexibility of creation, thereby improving the real-time projection effect.

[0089] At the same time, the virtual captured image is processed through an image mask to adjust the texture of the virtual three-dimensional object in the virtual captured image to obtain a target image, which effectively avoids adjusting the texture of the area other than the virtual three-dimensional object in the virtual captured image, ensures the integrity and accuracy of the area information in the virtual captured image other than the virtual three-dimensional object, makes the obtained target image more accurate, and thus improves the accuracy of the texture presented on the surface of the target object after projection.

[0090] In one embodiment, S110 may include: performing three-dimensional reconstruction on a real projection area where the target object is placed to obtain a virtual projection area, where the virtual projection area includes a virtual three-dimensional object corresponding to the target object.

[0091] The target object placed at any position in the real projection area can be a 3D model. The target object can be a 3D model with a texture on the surface or a 3D model without a texture on the surface (i.e., a white film model). The 3D model can be a human model, a building model, an animal model, a cultural relic model, etc. For example, an empty room with a bear model without a texture on the surface is placed in the real projection area.

[0092] The real projection area where the target object is placed can be reconstructed in three dimensions to obtain a three-dimensional virtual area as the virtual projection area corresponding to the real projection area. Since the target object is placed in the real projection area, a virtual three-dimensional object corresponding to the target object also exists in the constructed virtual projection area, that is, the target object exists in the virtual projection area in the form of a virtual three-dimensional object, and the virtual three-dimensional object is used as the virtual three-dimensional object corresponding to the target object.

[0093] The specific steps of three-dimensional reconstruction of the real projection area refer to the description of the above embodiment, which will not be repeated here. In the case where the real projection area is provided with a target object, the virtual environment constructed by the unity tool in the virtual system at least includes the virtual projection area, and the virtual projection area also includes a virtual three-dimensional object corresponding to the target object.

[0094] Correspondingly, before S130, the method may further include: acquiring an area in the virtual projection area where the virtual three-dimensional object is located as the target area.

[0095] That is to say, the area where the virtual three-dimensional object in the virtual projection area is located is taken as the target area, so that when the texture of the target area is adjusted, the texture of the virtual three-dimensional object is adjusted, so that when the obtained target captured image is projected onto the real projection area, the target object presents the texture of the virtual three-dimensional object in the target captured image, thereby realizing the texture adjustment of the target object and improving the immersive experience and projection effect of the target object.

[0096] In some embodiments, if there is one first virtual camera, there are multiple observation viewing angles, and a virtual projector is set at each observation viewing angle; then S120 includes: for each observation viewing angle, the first virtual camera is set at the observation viewing angle in the virtual projection area, and the virtual projection area is photographed by the first virtual camera to obtain a virtual captured image under the observation viewing angle; accordingly, S130 includes: adjusting the texture of the target area in the virtual captured image under each observation viewing angle to obtain the target image under each observation viewing angle; accordingly, S140 includes: projecting the target image under each observation viewing angle into the virtual projection area by a virtual projector located at each observation viewing angle.

[0097] That is to say, when there are multiple observation perspectives and there is only one first virtual camera, the virtual captured images at each observation perspective can be obtained by changing the position of the first virtual camera, and the texture of the target area (that is, the virtual three-dimensional object) in each virtual captured image can be adjusted to obtain the target image at each observation perspective, and then the target image at each observation perspective is projected onto the virtual projection area by a virtual projector located at each observation perspective.

[0098] In addition, it is worth mentioning that multiple observation perspectives are set around the virtual three-dimensional object, so that the virtual captured images obtained by the first virtual camera under different observation perspectives can present the complete form of the virtual three-dimensional object, so that the texture of the virtual three-dimensional object in the target image obtained under different observation perspectives can completely cover the virtual three-dimensional object, thereby achieving precise adjustment of the surface texture of the virtual three-dimensional object, and then when the target captured image is projected into the real projection area, there is no white edge on the surface texture of the target object, and the texture of the target object surface is more complete, thereby achieving the purpose of improving the projection effect.

[0099] Furthermore, in order to enable the target object in the real projection area to fully present the texture of the target image from multiple angles, multiple projection viewing angles can also be set (each projection viewing angle can be consistent with the aforementioned observation viewing angle, and the projection viewing angle can also include a projection distance and a projection direction, the projection distance can refer to the distance between the real projector and the target area, and the projection direction can refer to the direction pointed by the optical axis of the real projector). Each projection viewing angle is set around the target object, and each projection viewing angle is set with a real projector and a second virtual camera, so that the target image taken by the second virtual camera under different projection viewing angles can completely cover the virtual three-dimensional object, so that when the real projector under different projection viewing angles projects the target image, the surface of the target object can fully reflect the texture indicated by the target image, thereby improving the projection effect, the immersion of the user experience and the flexibility of creation.

[0100] For example, the observation angle includes the front observation angle and the rear observation angle, and the projection time also includes the front projection angle and the rear projection angle. The texture of the virtual three-dimensional object in the target image m1 corresponding to the front observation angle is texture s1, and the texture of the virtual three-dimensional object in the target image m2 corresponding to the rear observation angle is texture s2. The target image m1 is projected to the virtual projection area by the virtual projector of the front observation angle, and the target image m2 is projected to the virtual projection area by the virtual projector of the rear observation angle. The virtual projection area is photographed by the second virtual camera of the front projection angle to obtain the target photographed image mp1, and the virtual projection area is photographed by the second virtual camera of the rear projection angle to obtain the target photographed image mp2. The real projector of the front projection angle projects mp1 to the real projection area, and the real projector of the rear projection angle projects mp2 to the real projection area. At this time, the front side of the target object displays the texture s1, and the back side of the target object displays the texture s2.

[0101] For example, for a virtual image captured from the front observation angle, Figure 4 As shown, based on the preset texture image including different textures, the target image obtained is as follows Figure 5-7 shown.

[0102] For example, after the target captured images at each projection angle are projected onto the real projection area by the real projector at each projection angle, the target objects observed at different angles are as follows: Figure 8-Figure 10 As shown, the texture of the target object observed at different angles is different.

[0103] In another embodiment, there are multiple observation viewing angles; a virtual projector and a first virtual camera are set at each observation viewing angle; S120 may include: photographing the virtual projection area by the first virtual camera located at each observation viewing angle to obtain a virtual photographed image at each observation viewing angle; correspondingly, S130 includes: adjusting the texture of the target area in the virtual photographed image at each observation viewing angle to obtain the target image at each observation viewing angle; correspondingly, S140 includes: projecting the target image at each observation viewing angle onto the virtual projection area by the virtual projector located at each observation viewing angle.

[0104] That is to say, when there are multiple observation perspectives and each observation perspective corresponds to a first virtual camera, the virtual captured images at each observation perspective can be directly obtained through each first virtual camera, and the texture of the target area (that is, the virtual three-dimensional object) in each virtual captured image can be adjusted to obtain the target image at each observation perspective, and then the target image at each observation perspective is projected onto the virtual projection area through a virtual projector located at each observation perspective.

[0105] Reference Fig.11 , Fig.11 A block diagram of a projection control device according to an embodiment of the present application is shown, wherein the device 600 includes:

[0106] The reconstruction module 610 is used to perform three-dimensional reconstruction on the real projection area where the target object is placed to obtain a virtual projection area; the virtual projection area includes a virtual three-dimensional object corresponding to the target object;

[0107] The shooting module 620 is used to shoot the virtual projection area through virtual cameras of multiple observation angles to obtain a virtual shooting image at each observation angle; the position of the virtual camera at the observation angle is determined based on the position of the real projector at the observation angle;

[0108] The texture adjustment module 630 is used to adjust the texture of the virtual three-dimensional object in the virtual captured image at each observation angle to obtain a target image at each observation angle;

[0109] The control module 640 is used to project the target image at each observation angle into the real projection area through the real projector at each observation angle, so that the texture presented on the surface of the target object at each observation angle corresponds to the texture of the virtual three-dimensional object in the target image corresponding to each observation angle.

[0110] Optionally, the texture adjustment module 630 is also used to generate a corresponding image mask for the virtual captured image at each observation perspective; the image mask corresponding to the virtual captured image is used to indicate the area outside the virtual three-dimensional object in the virtual captured image; for each observation perspective, based on the virtual captured image at the observation perspective and the corresponding image mask, the texture of the virtual three-dimensional object in the virtual captured image at the observation perspective is adjusted to obtain the target image at the observation perspective.

[0111] Optionally, the texture adjustment module 630 is also used to fuse the virtual captured image under the observation perspective and the image mask corresponding to the virtual captured image under the observation perspective, so as to remove the pixel area corresponding to the virtual three-dimensional object in the virtual captured image under the observation perspective, and obtain a first fused image under the observation perspective; to fuse the image mask corresponding to the virtual captured image under the observation perspective and the preset texture image, so as to generate a texture in the area where the virtual three-dimensional object is located in the image mask corresponding to the virtual captured image under the observation perspective, and obtain a second fused image under the observation perspective; and to fuse the first fused image under the observation perspective and the second fused image to obtain a target image under the observation perspective.

[0112] Optionally, the reconstruction module 610 is also used to obtain a real captured image of the real projection area through a real camera when the target real projector projects a target projection image onto the real projection area where the target object is placed; and to perform three-dimensional reconstruction of the real projection area based on the number of real cameras and the real captured image to obtain a virtual projection area.

[0113] Optionally, the reconstruction module 610 is also used to determine the point cloud corresponding to the real projection area based on the number of real cameras and the real captured images; perform at least one of outlier removal processing, filtering processing and hole filling processing on the point cloud to obtain an intermediate point cloud; and perform triangle facet filtering processing on multiple triangular facets generated according to the intermediate point cloud to obtain a virtual projection area.

[0114] Optionally, the reconstruction module 610 is further used to determine the point cloud corresponding to the real projection area according to the relative relationship between the real camera and the target real projector, the real captured image and the target projection image if the number of the real camera is one.

[0115] Optionally, the reconstruction module 610 is further used to determine the point cloud corresponding to the real projection area according to the relative relationship between the real cameras and the real captured images of each real camera if the number of the real cameras is greater than one.

[0116] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0117] In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or modules, which may be electrical, mechanical or other forms.

[0118] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0119] Reference Figure 6 , Figure 6 A block diagram of an electronic device of another embodiment of the present application is shown. The electronic device 1200 may include one or more of the following components: a processor 1210, a memory 1220, and one or more application programs, wherein the one or more application programs may be stored in the memory 1220 and configured to be executed by the one or more processors 1210, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0120] The processor 1210 may include one or more processing cores. The processor 1210 uses various interfaces and lines to connect the various parts of the entire electronic device 1200, and executes various functions and processes data of the electronic device 1200 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1220, and calling data stored in the memory 1220. Optionally, the processor 1210 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1210, but may be implemented separately through a communication chip.

[0121] The memory 1220 may include a random access memory (RAM) or a read-only memory (ROM). The memory 1220 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 1220 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the terminal 1200 during use.

[0122] Please refer to Figure 7 , Figure 7 The structure block diagram of a computer-readable storage medium in an embodiment of the present application is shown. The computer-readable medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0123] The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method step of the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A projection control method, characterized in that: The method comprises: Perform three-dimensional reconstruction on the real projection area to obtain a virtual projection area; The virtual projection area is photographed by a first virtual camera located at an observation angle to obtain a virtual photographed image corresponding to the observation angle; Adjusting the texture of the target area in the virtual captured image under the observation angle to obtain the target image under the observation angle; Projecting the target image at the observation angle into the virtual projection area by a virtual projector located at the observation angle; The virtual projection area on which the target image is projected is photographed by a second virtual camera to obtain a target photographed image; the position of the second virtual camera is determined based on the position of the real projector; The target shot image is projected into the real projection area by the real projector.

2. The method according to claim 1, characterized in that The adjusting the texture of the target area in the virtual captured image under the observation angle to obtain the target image under the observation angle includes: Acquire an image mask corresponding to the target area under the observation angle; the image mask corresponding to the virtual captured image is used to indicate the area where the target area is located in the virtual captured image; Based on the virtual captured image under the observation viewing angle and the image mask corresponding to the virtual captured image under the observation viewing angle, the texture of the target area in the virtual captured image under the observation viewing angle is adjusted to obtain the target image under the observation viewing angle.

3. The method according to claim 2, characterized in that The adjusting the texture of the target area in the virtual captured image under the observation angle based on the virtual captured image under the observation angle and the image mask corresponding to the virtual captured image under the observation angle to obtain the target image under the observation angle includes: Acquire texture description information for the target object; Through the artificial intelligence content generation model, based on the texture description information and the image mask corresponding to the virtual captured image under the observation angle, the texture of the pixel area where the target area is located in the virtual captured image under the observation angle is adjusted to the texture indicated by the texture description information, so as to obtain the target image under the observation angle.

4. The method according to claim 1, characterized in that The real projection area is reconstructed in three dimensions to obtain the virtual projection area, including: Performing three-dimensional reconstruction on a real projection area where the target object is placed to obtain a virtual projection area, wherein the virtual projection area includes a virtual three-dimensional object corresponding to the target object; Before adjusting the texture of the target area in the virtual captured image under the observation angle, the method further includes: The area where the virtual three-dimensional object is located in the virtual projection area is obtained as the target area.

5. The method according to claim 4, characterized in that There is one first virtual camera, there are multiple observation angles, and a virtual projector is set under each observation angle; The step of photographing the virtual projection area by a first virtual camera located at an observation angle to obtain a virtual photographed image corresponding to the observation angle includes: For each of the observation angles, the first virtual camera is set at the observation angle in the virtual projection area, and the virtual projection area is photographed by the first virtual camera to obtain a virtual photographed image at the observation angle; The adjusting the texture of the target area in the virtual captured image under the observation angle to obtain the target image under the observation angle includes: Adjusting the texture of the target area in the virtual captured image at each observation viewing angle to obtain the target image at each observation viewing angle; The step of projecting the target image under the observation angle to the virtual projection area by a virtual projector located at the observation angle comprises: The target image at each observation viewing angle is projected into the virtual projection area through a virtual projector located at each observation viewing angle.

6. The method according to claim 4, characterized in that There are multiple observation viewing angles; a virtual projector and a first virtual camera are set under each observation viewing angle; The step of photographing the virtual projection area by a first virtual camera located at an observation angle to obtain a virtual photographed image corresponding to the observation angle includes: The virtual projection area is photographed by a first virtual camera corresponding to each observation angle to obtain a virtual photographed image under each observation angle; The adjusting the texture of the target area in the virtual captured image under the observation angle to obtain the target image under the observation angle includes: Adjusting the texture of the target area in the virtual captured image at each observation viewing angle to obtain the target image at each observation viewing angle; The step of projecting the target image under the observation angle to the virtual projection area by a virtual projector located at the observation angle comprises: The target image at each observation viewing angle is projected into the virtual projection area respectively through a virtual projector located at each observation viewing angle.

7. The method according to any one of claims 4 to 6, characterized in that: The multiple observation viewing angles are arranged around the target area.

8. A projection control device, characterized in that: The device comprises: A reconstruction module is used to perform three-dimensional reconstruction of the real projection area to obtain a virtual projection area; A first shooting module, used for shooting the virtual projection area through a first virtual camera located at an observation angle to obtain a virtual shot image corresponding to the observation angle; A texture adjustment module, used to adjust the texture of the target area in the virtual captured image under the observation angle to obtain the target image under the observation angle; A projection module, configured to project a target image under the observation viewing angle into the virtual projection area through a virtual projector located at the observation viewing angle; A second shooting module is used to shoot a virtual projection area on which a target image is projected through a second virtual camera to obtain a target shot image; the position of the second virtual camera is determined based on the position of the real projector; A control module is used to project the target captured image onto the real projection area through the real projector.

9. An electronic device, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images