Video stitching method, device and equipment in three-dimensional scene

Abstract

The application discloses a video splicing method, device and equipment in a three-dimensional scene. The method comprises the following steps: acquiring an initial three-dimensional model of a target scene and an image of a first region in the target scene collected by an image collection device; performing texture mapping on the initial three-dimensional model according to scene environment attribute information in the image to obtain a reference three-dimensional model; splicing the reference three-dimensional model and the image to obtain an initial splicing model; determining other scene attribute information in the target scene according to the image, and adjusting the initial splicing model according to the other scene attribute information to obtain a target splicing model. The application analyzes and identifies the environment attribute of the image, dynamically generates a model texture consistent with the image picture, and solves the abrupt splicing problem caused by inconsistent environment textures by using adaptive texture mapping. Furthermore, the initial splicing model after texture mapping is adjusted according to other scene attributes, and the virtual-real combination visual effect is further improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, and device for video stitching in a three-dimensional scene. Background Technology

[0002] In digital twin and visualization systems, in order to improve the realism of the visualized scene, cameras are used to capture video images of certain areas in the visualized display scene. The real-time captured video images are then stitched into the 3D scene model. That is, the area in the 3D scene model corresponding to the camera's viewpoint uses the real-time video image, while the area that other cameras cannot capture uses the 3D scene model. Therefore, in the visualization process, the stitching effect between the real-time video image and the 3D scene model affects the overall visualization effect.

[0003] Since the 3D scene model is pre-built, the visual effect of the video screen area and the 3D scene model may be inconsistent due to the dynamic changes in the perspective of the real-time video screen (such as the drone diving and the camera rotating) and the changes in environmental factors in reality (such as season, weather and lighting), resulting in the screen area being fragmented or spliced ​​abruptly in the model.

[0004] Existing technologies typically employ fixed perspectives or simple image stitching methods, which cannot dynamically adjust 3D scene models to adapt to changes in video perspective and environment, thus affecting the visualization effect of combining virtual and real elements. Summary of the Invention

[0005] This invention provides a method, apparatus, and device for video stitching in a three-dimensional scene to solve the problem of poor visual effects when stitching actual scene video images into a three-dimensional model.

[0006] According to one aspect of the present invention, a video stitching method in a three-dimensional scene is provided, comprising:

[0007] Acquire the initial 3D model of the target scene and the image of the first region in the target scene acquired by the image acquisition device;

[0008] Based on the scene environment attribute information in the image, the initial 3D model is texture-mapped to obtain a reference 3D model;

[0009] The reference 3D model and the image are stitched together to obtain an initial stitched model;

[0010] Based on the image, other scene attribute information in the target scene is determined, and the initial stitching model is adjusted based on the other scene attribute information to obtain the target stitching model.

[0011] According to another aspect of the present invention, a video stitching device in a three-dimensional scene is provided, comprising:

[0012] The model and image acquisition module is used to acquire the initial 3D model of the target scene and the image of the first region in the target scene acquired by the image acquisition device;

[0013] The model texture mapping module is used to perform texture mapping on the initial 3D model based on the scene environment attribute information in the image to obtain a reference 3D model.

[0014] The stitching module is used to stitch the reference 3D model and the image to obtain an initial stitched model;

[0015] The stitching model adjustment module is used to determine other scene attribute information in the target scene based on the image, and adjust the initial stitching model based on the other scene attribute information to obtain the target stitching model.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the video stitching method in a three-dimensional scene according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the video stitching method in a three-dimensional scene as described in any embodiment of the present invention.

[0021] The technical solution of this embodiment analyzes and identifies the environmental attributes of the image, dynamically generates a model texture consistent with the image, and adopts adaptive texture mapping to ensure visual consistency between the model and the image when performing virtual-real stitching, thus solving the problem of abrupt stitching caused by inconsistent environmental textures. Furthermore, it adjusts other scene attributes of the initial stitching model after texture mapping, further improving the visualization effect of virtual-real combination and enhancing the realism of virtual-real combination, providing more technical support for digital twin and visualization systems.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a video stitching method in a three-dimensional scene according to an embodiment of the present invention;

[0025] Figure 2 This is a rendering of the video stitching effect in a three-dimensional scene according to an embodiment of the present invention;

[0026] Figure 3 This is a flowchart of another video stitching method in a three-dimensional scene provided by an embodiment of the present invention;

[0027] Figure 4 This is a flowchart of another video stitching method in a three-dimensional scene provided by an embodiment of the present invention;

[0028] Figure 5 This is a structural schematic diagram of a video stitching device in a three-dimensional scene according to an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the video stitching method in a three-dimensional scene according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "candidate," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Figure 1 This invention provides a flowchart of a video stitching method in a three-dimensional scene. This embodiment is applicable to stitching real-world video footage into a three-dimensional model. The method can be executed by a video stitching device in the three-dimensional scene, which can be implemented in hardware and / or software. This video stitching device can be configured on a server with computing and communication capabilities. Figure 1 As shown, the method includes:

[0033] S110. Obtain the initial 3D model of the target scene and the image of the first region in the target scene acquired by the image acquisition device.

[0034] The target scene refers to the area scene that needs to be visualized and digitally twinned, such as a street scene. This involves creating a digital twin of the street scene and then visualizing it. The initial 3D model refers to the 3D display model determined based on the actual scene layout in the target scene. For example, a rendered image of the model in the target scene is acquired, and a 3D model is rendered based on this rendered image to obtain the initial 3D model corresponding to the target scene. In other words, the initial 3D model is constructed based on the rendered image. Since model construction requires a certain amount of work, the initial 3D model of the target scene is generally not changed arbitrarily after it is completed; that is, it is always used as the basis for subsequent visualization displays.

[0035] Image acquisition equipment refers to devices that acquire real-time images of a first area within a target scene, such as drones equipped with cameras or surveillance cameras deployed in the first area. The first area is a portion of the target scene. Because the target scene is large, the image acquisition equipment cannot capture the entire area. Therefore, when visualizing the target scene, the area containing key information is designated as the first area. The first area is visualized using images acquired by the image acquisition equipment, while other areas of the target scene besides the first area are displayed using an initial 3D model.

[0036] Specifically, a model rendering image containing all information within the target scene is acquired. Based on this rendering image, the target scene is rendered to obtain an initial 3D model. An image acquisition device is used to acquire real-time images of a first region within the target scene. These real-time images of the first region are then stitched together with the initial 3D model. This allows for visualization of the target scene, where the first region displays the real-time image, while other regions display the 3D model, thus improving the visual effect of the visualization. For example... Figure 2 The image shown is a rendering of video stitching in a three-dimensional scene according to an embodiment of the present invention. The target scene is within the entire area shown in the rendering. The area within the red box is the first area, which is a road scene and contains key information. Therefore, the content in the first area is displayed using real-time video captured by a camera. The other areas outside the red box are fixed landscape scenes, which contain less key information. Therefore, they are displayed using 3D models.

[0037] S120. Based on the scene environment attribute information in the image, perform texture mapping on the initial 3D model to obtain a reference 3D model.

[0038] Since the initial 3D model is determined based on the model rendering image during construction, the environmental attributes in the initial 3D model are determined based on the environmental attributes of the model rendering image. However, the image is captured in real time by an image acquisition device, and the actual environmental factors in the target scene are changing and are not the same as the environmental factors when the initial 3D model was constructed. Therefore, in order to improve the environmental consistency between the 3D model and the image, the texture of the initial 3D model is adjusted.

[0039] Specifically, scene environment attribute information is extracted from the image. This information includes at least seasonal and weather information, and further includes lighting information. A corresponding texture image is generated based on this information, matching the texture in the image. This texture image is then rendered as an environment map onto the initial 3D model, resulting in a textured reference 3D model.

[0040] Optionally, since the consistency of the stitching edge region is particularly important when stitching the actual image with the 3D model, the scene environment attribute information also includes object feature information of the image edge region. This ensures that the model texture obtained based on the scene environment attribute information includes the object texture of the stitching edge region. This texture is then mapped onto the initial 3D model, making the edge region of the reference 3D model that is stitched with the image consistent with the image, thus improving the visualization effect of the stitching result. For example, if the initial 3D model does not have trees in the stitching edge region when it is constructed, but trees are present in this region when the image is acquired, the corresponding scene environment attribute information is determined based on the tree features, so that the generated model texture includes the tree texture. During texture mapping, the tree texture is mapped onto the stitching edge region based on the position information, improving the stitching effect.

[0041] S130. The reference 3D model and the image are stitched together to obtain the initial stitched model.

[0042] After rendering the texture in the reference 3D model, the reference 3D model and the image are stitched together to obtain the initial stitched model. In the initial stitched model, the first area is displayed as the image, and the other areas are the images of the reference 3D model.

[0043] For example, the boundary coordinates corresponding to the image are determined, and the corresponding boundary position in the reference 3D model is determined based on the boundary coordinates. That is, the corresponding position of the first region in the reference 3D model is determined, and the image is superimposed on the corresponding position in the reference 3D model. Since the texture of the reference 3D model is obtained based on the scene environment attribute information of the image, the environment of the image and the model in the initial stitching model is kept consistent, ensuring a more natural stitching effect and improving the visualization effect. Alternatively, after determining the corresponding position of the first region in the reference 3D model, a portion of the model at the corresponding position is deleted, and the image is used to fill that portion.

[0044] S140. Determine other scene attribute information in the target scene based on the image, and adjust the initial stitching model based on the other scene attribute information to obtain the target stitching model.

[0045] Other scene attribute information is used to represent visual effects influencing factors in the target scene other than environmental attributes. These include lighting parameters and shadow information. Specifically, lighting parameters represent the lighting information in the target scene at the time of image capture, while shadow information represents the visual or data features contained in the bright and dark areas formed by the interaction of light and objects in the target scene at the time of image capture.

[0046] Specifically, the types and corresponding features of other scene attribute information are determined, and other scene attribute information of the target scene is extracted from the image based on the features. The corresponding model attribute information in the model is determined based on the other scene attribute information of the target scene. The initial stitching model is adjusted based on the model attribute information to obtain the final target stitching model.

[0047] For example, shadow features are extracted from an image to obtain the scene shadow region. The model shadow region is then determined based on the scene shadow region, and the model shadow region is superimposed onto the initial stitching model to obtain the target stitching model. For instance, the extraction of shadow features and the generation of the model shadow region can be determined based on pre-trained shadow extraction and shadow generation models.

[0048] The technical solution of this embodiment analyzes and identifies the environmental attributes of the image, dynamically generates a model texture consistent with the image, and adopts adaptive texture mapping to ensure visual consistency between the model and the image when performing virtual-real stitching, thus solving the problem of abrupt stitching caused by inconsistent environmental textures. Furthermore, it adjusts other scene attributes of the initial stitching model after texture mapping, further improving the visualization effect of virtual-real combination and enhancing the realism of virtual-real combination, providing more technical support for digital twin and visualization systems.

[0049] Figure 3 This is a flowchart illustrating another video stitching method in a three-dimensional scene provided by an embodiment of the present invention. This embodiment further refines the texture mapping in the above embodiments. Figure 3 As shown, the method includes:

[0050] S310. Obtain the initial 3D model of the target scene and the image of the first region in the target scene acquired by the image acquisition device.

[0051] S320. Determine the scene environment attribute information in the target scene based on the image.

[0052] Scene environment attribute information is used to represent the natural environmental features of the target scene. For the same target scene, the visual impact of taking photos at different times is mostly concentrated on changes in the natural scenery. Therefore, scene environment attribute information includes at least seasonal information and weather information. Seasonal and weather information can be extracted and determined from the natural landscape features in the image.

[0053] Specifically, sample images and their corresponding environmental attribute information are pre-determined. An environmental attribute extraction network is trained based on these sample images and environmental attribute information to learn environmental attribute features in the images and extract environmental attribute information from them. The input to the environmental attribute extraction network is the image, and the output is the environmental attribute vector corresponding to each pixel in the image. For example, an environmental attribute extraction function E(x) is defined, where x is the pixel value of each pixel in the image: e = E(x), and the output is the environmental attribute vector e, which includes seasonal, weather, and other feature information for each location point in the first region of the target scene.

[0054] S330. Generate model textures based on scene environment attribute information, and map the model textures onto the initial 3D model to obtain a reference 3D model.

[0055] Based on the scene environment attribute information, a model texture consistent with the image is dynamically generated, and the generated model texture is mapped onto the initial 3D model using texture mapping technology to obtain a reference 3D model whose model texture is similar to the image texture.

[0056] Specifically, the sample environment attribute information and the corresponding sample model texture are pre-determined. A texture generation network is trained based on this information to learn the features between environment attributes and textures. Based on the extracted environment attributes, a 3D model texture consistent with the image is dynamically generated. The input to the texture generation network is the scene environment attribute information corresponding to each point in the image. By learning the features between environment attributes and textures, and the correspondence between points in the image and the 3D model, the output is the model texture corresponding to each point in the model. For example, a texture generation function G(e) is defined to generate the model texture based on the environment attribute vector e: Texture = G(e). For instance, a GAN or CNN network can be used to generate high-quality model textures, ensuring that the model texture is consistent with the image environment.

[0057] In one feasible embodiment, generating model textures based on scene environment attribute information includes:

[0058] Determine the device attribute parameters of the image acquisition device, and determine the perspective transformation relationship between the image and the initial 3D model based on the device attribute parameters;

[0059] Based on the perspective transformation relationship, perform perspective transformation on the three-dimensional points in the second region of the initial three-dimensional model to obtain the corresponding texture coordinates of the second region;

[0060] Generate model textures corresponding to each texture coordinate based on scene environment attribute information.

[0061] Among them, the device attribute parameters are used to represent the acquisition perspective of the image acquisition device, so as to determine the relationship between the image coordinates and the world coordinates based on the acquisition perspective of the image acquisition device. Since the initial 3D model is established in the world coordinate system, the determined relationship between the image coordinates and the world coordinates is the perspective transformation relationship between the image and the initial 3D model.

[0062] Since the image acquisition device captures two-dimensional images, while the model is three-dimensional, when generating model textures based on the scene environment attribute information of the two-dimensional images, it is necessary to establish a correspondence between the image and the three-dimensional model in terms of dimensions to ensure that the generated model texture matches the position information in the three-dimensional model.

[0063] Specifically, the motion trajectory and viewpoint parameters of the image are determined based on feature point matching or optical flow. Device attribute parameters are then determined based on these parameters, and the relationship between image coordinates and world coordinates is determined based on these device attribute parameters, serving as the perspective transformation relationship between the image and the initial 3D model. Based on this perspective transformation relationship, perspective transformation is performed on the 3D points in the second region of the initial 3D model to obtain the corresponding 2D texture coordinates for that region. The model texture corresponding to each 2D texture coordinate in the second region is determined based on the scene environment attribute information of each point in the first region of the target scene. The second region can be the entire region of the initial 3D model, or, to improve computational efficiency and reduce the workload of perspective transformation, it can be any region in the initial 3D model other than the first region. That is, perspective transformation is performed on the 3D points in other regions to determine their model textures, while the texture in the first region can be the texture from the image itself, thus reducing computational load.

[0064] For example, the perspective transformation relationship between the image and the initial 3D model is determined as the perspective transformation matrix P based on the device attribute parameters. The 3D coordinates X of each 3D point in the second region of the initial 3D model are mapped to the image plane through perspective transformation: i.e., x = P·X, where x is the coordinate of the 2D image plane. Then, texture mapping is performed, defining a texture mapping function T(μ, υ) to map the generated model texture to the surface of the initial 3D model: T(μ, υ) = Texture(μ, υ, e). Here, (μ, υ) are the texture coordinates of the second region of the initial 3D model after perspective transformation, and e is the scene environment attribute information of the image, including the environment attribute vector of each point in the first region. Optionally, the texture mapping effect can be optimized by interpolation algorithms (such as bilinear interpolation).

[0065] Based on the changes in the viewing angle of the image acquired by the image acquisition device, the scene environment attribute information is adaptively adjusted, and the perspective transformation matrix P and texture mapping function T(μ,υ) are updated in real time to ensure the visual consistency between the reference 3D model after texture mapping and the image.

[0066] This embodiment dynamically adjusts the perspective relationship of the initial 3D model based on the image viewpoint parameters, and maps the generated texture onto the model surface, thereby improving the positional accuracy and matching degree of texture mapping.

[0067] In one feasible embodiment, the device attribute parameters include at least device position information, rotation angle information, and intrinsic parameter matrix;

[0068] Determine the perspective transformation relationship between the image and the initial 3D model based on device attribute parameters, including:

[0069] The perspective transformation matrix is ​​determined based on the device position information, rotation angle information, and intrinsic parameter matrix, serving as the perspective transformation relationship between the image and the initial 3D model.

[0070] Among them, the device position information refers to the position coordinates of the image acquisition device in the world coordinate system, the rotation angle information is used to represent the orientation information of the image acquisition device, the rotation angle information is the rotation matrix of the image acquisition device, and the intrinsic parameter matrix is ​​used to represent the intrinsic parameters of the image acquisition device, which includes the focal length information of the image acquisition device and the coordinate information of the image center point.

[0071] Based on the image information acquired by the image acquisition device and the sensor information (such as IMU, GPS, LiDAR) on the device, the device position information, rotation angle information, focal length, field of view and image center point coordinates are collected in real time.

[0072] The perspective transformation matrix P is determined according to the following formula, representing the perspective transformation relationship between the image and the initial 3D model: P = K[R|t], where K represents the intrinsic parameter matrix, denoted as... (c x ,c y ) represents the coordinates of the image center point, f represents the focal length, R represents the rotation matrix, and t represents the device position information, t = [t x ,t y ,t z ].

[0073] Optionally, the motion trajectory and viewpoint parameters of the image acquisition device can be updated in real time through feature point matching or optical flow, and the device position information t, rotation matrix R, and perspective transformation matrix P can be adaptively and promptly adjusted in real time.

[0074] This embodiment determines device attribute information by acquiring the perspective parameters of the image from the image acquisition device in real time, thereby improving the accuracy of perspective transformation relationship determination and texture mapping accuracy.

[0075] S340. The reference 3D model and the image are stitched together to obtain the initial stitched model.

[0076] In one feasible embodiment, S340 includes:

[0077] The image stitching weight of the first pixel is determined based on the pixel value of the first pixel in the fourth region of the image.

[0078] The first stitching value of the first pixel is determined by multiplying the pixel value of the first pixel in the image with the image stitching weight;

[0079] Determine the second pixel point in the reference 3D model that corresponds to the first pixel point;

[0080] The model stitching weight of the second pixel is determined based on the image stitching weight of the first pixel, and the second stitching value of the second pixel is determined based on the product of the pixel value of the second pixel in the reference 3D model and the model stitching weight.

[0081] Based on the sum of the first and second stitching values, determine the pixel value of the pixel corresponding to the second pixel in the initial stitching model.

[0082] When stitching an image and a 3D model, the image can be directly superimposed on the corresponding area of ​​the 3D model. However, this superimposition process involves the fusion of pixel values ​​between the image and the 3D model. To improve the stitching effect, the fourth region is set as the entire area of ​​the image, corresponding to the first region in the target scene. This means that the entire image area is fused with the reference 3D model to obtain the stitching result. On the other hand, to improve stitching efficiency, the first region is divided into two parts: an edge region and a center region. In the initial stitching model, the center region directly uses the image information at the corresponding position, while the edge region uses the fusion effect of the image and the model. At this time, the fourth region is the edge region of the image, i.e., the edge region of the first region. The edge region is the contact area between the 3D model and the image. The size of the edge region can be determined according to the actual situation and is not limited here.

[0083] The following formula is used to stitch the image and the reference 3D model together: I f =α·I video +(1-α)·I m Among them, I f I represents the pixel value of the pixel corresponding to the second pixel in the initial stitching model. video I represents the pixel value of the first pixel in the image. mThis represents the pixel value of the second pixel corresponding to the first pixel in the reference 3D model. The correspondence between the first and second pixels is determined through perspective transformation. α represents the image stitching weight. Since the closer the stitched result is to the image, the more realistic the visualization effect of the stitched model, the pixel value of the first pixel is positively correlated with its corresponding image stitching weight. That is, the larger the pixel value of the first pixel, the larger its image stitching weight.

[0084] This embodiment fuses and stitches real-time images with a texture-mapped reference 3D model to eliminate stitching marks in boundary areas and improve the stitching effect. Furthermore, during the stitching process, the corresponding image stitching weights are dynamically adjusted based on the pixel values ​​of the images, that is, the degree of influence of the image on the stitching effect is dynamically adjusted to improve the visual consistency of the stitched image in terms of combining the virtual and real elements.

[0085] Optionally, a Poisson stitching algorithm can be used to eliminate the boundary stitching marks between the image and the 3D model. For example, the following formula can be used to stitch the image and the reference 3D model: in, For the Laplace operator.

[0086] S350. Determine other scene attribute information in the target scene based on the image, and adjust the initial stitching model based on the other scene attribute information to obtain the target stitching model.

[0087] The technical solution of this embodiment, through adaptive perspective and texture mapping technology, ensures visual consistency between the 3D model and the image when performing virtual-real stitching, solves the problem of abrupt stitching caused by inconsistent environmental textures, and improves the visual effect of stitching real-time images and 3D models.

[0088] Figure 4 This is a flowchart of another video stitching method in a three-dimensional scene provided by an embodiment of the present invention. This embodiment further refines the step of "determining other scene attribute information in the target scene based on the image, and adjusting the initial stitching model based on the other scene attribute information" in the above embodiment. Figure 4 As shown, the method includes:

[0089] S410. Obtain the initial 3D model of the target scene and the image of the first region in the target scene acquired by the image acquisition device.

[0090] S420. Based on the scene environment attribute information in the image, perform texture mapping on the initial 3D model to obtain a reference 3D model.

[0091] S430. The reference 3D model and the image are stitched together to obtain the initial stitched model.

[0092] S440. Analyze the image based on the pre-trained illumination estimation model to obtain scene illumination parameter information, and use the scene illumination parameter information as other scene attribute information.

[0093] The scene lighting parameter information is used to represent the lighting attribute features of the target scene in the image. The scene lighting parameter information includes scene light source direction information, scene lighting intensity information, and scene ambient light color information.

[0094] Specifically, sample images and their corresponding illumination parameters are predetermined. For example, multiple sets of illumination parameters are pre-set, and corresponding sample images are generated based on each set of illumination parameters. An illumination estimation model is then trained based on the sample images and their corresponding illumination parameters to learn the illumination features in the images. The input to the illumination estimation model is the image, and the output is the scene illumination parameters of that image.

[0095] For example, lighting information is extracted from video footage. A lighting estimation model is used to analyze the image and extract the following lighting parameters: scene light source direction information L. v =[L vx L vy L vz ]: Used to represent the direction vector of the light source in the image; scene illumination intensity information I v : Used to represent the light intensity value in an image; scene ambient light color information C v =[C vx C vy C vz ]: Used to represent the ambient light color (RGB value) in an image. For example, the illumination estimation model H(x) can be defined using the formula: H(x) = L v I v C v .

[0096] S450. Adjust the model lighting parameters in the initial stitching model according to the scene lighting parameters to obtain the target stitching model.

[0097] The initial model lighting parameters are determined based on the model rendering image during model construction. The initial model lighting parameters are then adjusted based on the scene lighting parameters to obtain the adjusted model lighting parameters. Finally, the target model is obtained based on the adjusted model lighting parameters.

[0098] Specifically, scene weights and model weights are determined, and the adjusted model lighting parameters are determined based on the weighted value of the scene lighting parameters and the initial model lighting parameters. Alternatively, the scene lighting parameters can be directly set to the adjusted model lighting parameters. The values ​​of scene weights and model weights can be determined based on the values ​​of the scene lighting parameters; for example, the values ​​of scene lighting parameters are positively correlated with the values ​​of scene weights, and the values ​​of scene weights are inversely correlated with the values ​​of model weights.

[0099] For example, the lighting settings of the 3D stitching model are dynamically adjusted based on the lighting parameters extracted from the image: the light source direction L of the stitching model is updated. m For L v Adjust the lighting intensity of the spliced ​​model I m For I v Set the ambient light color C of the spliced ​​model. m C v .

[0100] In one feasible embodiment, other scene attribute information in the target scene is determined based on the image, and the initial stitching model is adjusted based on the other scene attribute information, including:

[0101] Other scene attribute information is extracted from the image, including at least one of the following: scene shadow information, scene reflection information, and scene highlight information;

[0102] Determine the corresponding model attribute information based on other scene attribute information;

[0103] Other scene attribute information and corresponding model attribute information are concatenated to obtain concatenated attribute information, and the initial concatenated model is adjusted based on the concatenated attribute information.

[0104] Other scene attribute information refers to information about visual influencing factors in the target scene other than environmental texture. Other scene attribute information may include scene shadow information, which is used to represent the light and shadow information of the target scene in the image; other scene attribute information may also include scene reflection information, which is used to represent the information of objects with light reflection in the target scene in the image, such as object type, object size, etc.; other scene attribute information may also include scene highlight information, which is used to represent the result of light reflection in the target scene in the image, such as the size and brightness information of the light spot formed by light reflection.

[0105] If other scene attribute information includes scene shadow information, the scene shadow information is extracted from the image. This scene shadow information includes scene shadow regions, which represent the locations of areas with light and shadow in the target scene. Combining the scene shadow regions, scene light source direction information, and geometric information from the 3D model, a shadow mapping algorithm is used to generate model shadow regions, which serve as model attribute information corresponding to the scene shadow information. The scene shadow regions and model shadow regions are then stitched together to obtain a stitched shadow region. This stitched shadow region is then mapped onto the initial stitched model to obtain the initial stitched model with adjusted shadows.

[0106] For example, an image is analyzed using a shadow detection algorithm (such as a deep learning-based shadow segmentation network) to extract the scene's shadow region S. v Based on the scene's shadow area and the direction of the scene's light source L v Using the geometric information of the 3D model, a shadow mapping algorithm (such as a deep learning-based shadow generation network) is used to generate the model shadow region S of the 3D model. m The shadow area S in the image is... v The shadow area S of the 3D model m Perform stitching to ensure consistency of shaded areas in the stitched model: S f =α1·S v +(1-α1)·S m , among which, S f α1 represents the shadow area in the initial stitching model after shadow adjustment, and α1 represents the shadow stitching weight, which is dynamically adjusted according to the pixel value of the shadow area in the scene. The shadow stitching weight is positively correlated with the size of the corresponding pixel value to ensure that the visual effect of the initial stitching model after shadow adjustment is consistent with the image.

[0107] If other scene attribute information includes scene reflection information, the scene reflection regions and corresponding scene reflection coefficients are extracted from the image as scene reflection information. The scene reflection regions represent the locations of reflective areas in the target scene, and the scene reflection coefficients represent the reflectivity of each scene reflection region. Model reflection information is determined based on the scene reflection information and the geometric information in the 3D model, serving as the model attribute information corresponding to the scene reflection information. The model reflection information includes model reflection regions and corresponding model reflection coefficients. The model reflection regions represent the locations of reflective areas in the model, and the model reflection coefficients represent the reflectivity of each model reflection region. The scene reflection information and model reflection information are then stitched together to obtain stitched reflection information. This stitched reflection information is then mapped onto the initial stitched model, or onto the initial stitched model after shadow adjustment, to obtain the initial stitched model with reflection adjustment.

[0108] For example, an image is analyzed using a reflection detection algorithm (such as a deep learning-based reflection detection network) to extract the scene reflection region R1. v and scene reflection coefficient R2 v Based on the scene's reflection regions and the geometric information of the 3D model, a reflection mapping algorithm (such as a deep learning-based reflection generation network) is used to generate the model reflection region R1 of the 3D model. m and model reflection coefficient R2 m The scene reflection information R in the image frame v Model reflection information R of the 3D model m Perform splicing to ensure consistency of the reflection areas in the spliced ​​model: R f =α2·R v +(1-α2)·R m , where R f This represents the reflection information in the initial stitched model after reflection adjustment. R f Including R1 f , indicating the spliced ​​reflection area, and including R2 f α represents the splicing reflection coefficient, and α2 represents the reflection splicing weight, which is dynamically adjusted according to the pixel value of the scene's reflection area. The reflection splicing weight is positively correlated with the corresponding pixel value to ensure that the visual effect of the initial splicing model after reflection adjustment is consistent with the image.

[0109] If other scene attribute information includes scene specular information, the scene specular regions and corresponding scene specular coefficients are extracted from the image as scene specular information. The scene specular regions represent the positions of light-reflected spot areas in the target scene, and the scene specular coefficients represent the reflective brightness of each scene specular region. Model specular information is determined based on the scene specular information, scene light source direction information, and geometric information in the 3D model, serving as the model attribute information corresponding to the scene specular information. The model specular information includes model specular regions and corresponding model specular coefficients. The model specular regions represent the positions of light-reflected spot areas in the model, and the model specular coefficients represent the reflective brightness of each model specular region. The scene specular information and model specular information are then stitched together to obtain stitched specular information. This stitched specular information is then mapped onto the initial stitched model, or onto the initial stitched model after reflection adjustment, to obtain the specular-adjusted initial stitched model.

[0110] For example, an image is analyzed using a specular detection algorithm (such as a deep learning-based specular detection network) to extract the specular region K1 of the scene. v and scene specular value K2 vBased on the scene's specular regions and the geometric information of the 3D model, a specular mapping algorithm (such as a deep learning-based specular generation network) is used to generate the model's specular regions K1. m And model specular coefficient K2 m K, the scene highlight information in the image. v Specular information K of the 3D model m Perform stitching to ensure consistency in the highlight areas of the stitched model: K f =α3·K v +(1-α3)·K m , where K f This represents the specular information in the initial stitched model after specular adjustment. K f Including K1 f This indicates the splicing of highlight areas, including K2. f , represents the specular highlight coefficient, and α3 represents the specular highlight stitching weight, which is dynamically adjusted according to the pixel value of the specular region in the scene. The specular highlight stitching weight is positively correlated with the corresponding pixel value to ensure that the visual effect of the initial stitching model after specular adjustment is consistent with the image.

[0111] In this embodiment of the invention, the initial stitching model after texture adjustment is further adjusted by using scene shadow information, scene reflection information and scene highlight information in the image, so that the shadow vision, reflection vision and highlight vision in the adjusted 3D model are consistent with those in the image, thereby improving the visual effect of the stitching model.

[0112] In one feasible embodiment, the method further includes, before obtaining the target stitching model:

[0113] The illumination intensity error is determined based on the illumination intensity of each pixel in the third region of the image and the corresponding region in the initial stitching model.

[0114] The optimization objective is determined based on the light intensity error, and the light parameters in the initial stitching model are iteratively optimized.

[0115] The third region is the first region where the image is located, or, to improve computational efficiency, the remaining third region is the edge region where the image and the 3D model are stitched together.

[0116] Identify the third pixel points in the third region of the image, and the corresponding fourth pixel points in the initial stitching model. Determine the illumination intensity error based on the difference in illumination intensity between each third pixel point and the fourth pixel point. Specifically, the illumination intensity error E is calculated using the following formula: Among them, I v (i) represents the illumination intensity of the i-th third pixel, I m(i) represents the illumination intensity of the fourth pixel corresponding to the third pixel. The illumination intensity error is used to measure the difference in illumination between the image and the stitched 3D model.

[0117] Minimizing the illumination intensity error E is determined as the optimization objective. The illumination parameters in the stitching model are dynamically adjusted to match the image. Specifically, the optimization process is determined using the following formula: Iterative optimization is performed using algorithms such as gradient descent or Newton's method, updating the lighting parameters of the stitched 3D model iteratively until the error E converges. The iterative optimization stops at this point, and the corresponding lighting parameters become the lighting parameters in the final target stitched model. At this point, the lighting intensity error in the target stitched model is minimized compared to that in the image.

[0118] Lighting consistency optimization is a crucial step in ensuring a natural and realistic stitching effect between video footage and 3D models. Inconsistent lighting conditions (e.g., the video footage is overcast while the 3D model is in sunny weather) can lead to noticeable visual discontinuities in the stitched area. Dynamically adjusting the lighting parameters (such as shadows and reflections) of the 3D model based on the video's lighting conditions ensures a more natural stitching effect.

[0119] This embodiment measures the lighting difference between the image and the stitched 3D model by measuring the lighting intensity error, and then determines the lighting parameters in the target stitching model to achieve lighting consistency between the target stitching model and the image, so as to ensure that the stitching effect between the image and the target stitching model is natural and realistic.

[0120] The technical solution of this embodiment improves the visual effect of the stitching model by dynamically adjusting the relevant attributes of the texture after texture adjustment through the shadow attribute, reflection attribute and highlight attribute in the image, and ensures the visual consistency between the stitching model and the image.

[0121] This invention provides another video stitching method in a three-dimensional scene, specifically including:

[0122] Acquire the initial 3D model of the target scene and the image of the first region in the target scene acquired by the image acquisition device.

[0123] The scene environment attribute information in the target scene is determined based on the image; the scene environment attribute information includes at least: seasonal information and weather information; the device attribute parameters of the image acquisition device are determined, and the perspective transformation matrix is ​​determined based on the device position information, rotation angle information and intrinsic parameter matrix, as the perspective transformation relationship between the image and the initial 3D model; the 3D points in the second region of the initial 3D model are subjected to perspective transformation according to the perspective transformation relationship to obtain the corresponding texture coordinates of the second region; the model texture corresponding to each texture coordinate is generated according to the scene environment attribute information, and the model texture is mapped to the initial 3D model to obtain the reference 3D model.

[0124] The image stitching weight of the first pixel is determined based on the pixel value of the first pixel in the fourth region of the image; the first stitching value of the first pixel is determined based on the product of the pixel value of the first pixel and the image stitching weight; the second pixel in the reference 3D model corresponding to the first pixel is determined; the model stitching weight of the second pixel is determined based on the image stitching weight of the first pixel; the second stitching value of the second pixel is determined based on the product of the pixel value of the second pixel in the reference 3D model and the model stitching weight; the pixel value of the pixel corresponding to the second pixel in the initial stitching model is determined based on the sum of the first stitching value and the second stitching value, thus obtaining the initial stitching model.

[0125] The image is analyzed based on a pre-trained illumination estimation model to obtain scene illumination parameter information, which is then used as another scene attribute information. The scene illumination parameter information includes scene light source direction information, scene illumination intensity information, and scene ambient light color information. The illumination parameter information in the initial stitching model is adjusted according to the scene illumination parameter information to obtain the intermediate stitching model.

[0126] Other scene attribute information is extracted from the image, including scene shadow information, scene reflection information, and scene highlight information. Corresponding model attribute information is determined based on this information. The scene attribute information and the corresponding model attribute information are then stitched together to obtain stitched attribute information. The intermediate stitched model is then further adjusted sequentially based on this stitched attribute information to obtain a reference stitched model. Specifically, the intermediate stitched model is adjusted sequentially based on the stitched shadow information, stitched reflection information, and stitched highlight information to obtain the reference stitched model.

[0127] The illumination intensity error is determined based on the illumination intensity of each pixel in the third region of the image and the corresponding region in the reference stitching model; the optimization target is determined based on the illumination intensity error, and the illumination parameters in the reference stitching model are iteratively optimized to obtain the target stitching model.

[0128] For specific technical details in this embodiment, please refer to the above embodiments, which will not be repeated here.

[0129] Figure 5 This is a schematic diagram of a video stitching device in a three-dimensional scene provided by an embodiment of the present invention. Figure 5 As shown, the device includes:

[0130] The model and image acquisition module 510 is used to acquire the initial 3D model of the target scene and the image of the first region in the target scene acquired by the image acquisition device;

[0131] The model texture mapping module 520 is used to perform texture mapping on the initial three-dimensional model according to the scene environment attribute information in the image to obtain a reference three-dimensional model;

[0132] The stitching module 530 is used to stitch the reference 3D model and the image to obtain an initial stitched model;

[0133] The stitching model adjustment module 540 is used to determine other scene attribute information in the target scene based on the image, and adjust the initial stitching model based on the other scene attribute information to obtain the target stitching model.

[0134] The technical solution of this embodiment analyzes and identifies the environmental attributes of the image, dynamically generates a model texture consistent with the image, and adopts adaptive texture mapping to ensure visual consistency between the model and the image when performing virtual-real stitching, thus solving the problem of abrupt stitching caused by inconsistent environmental textures. Furthermore, it adjusts other scene attributes of the initial stitching model after texture mapping, further improving the visualization effect of virtual-real combination and enhancing the realism of virtual-real combination, providing more technical support for digital twin and visualization systems.

[0135] Optional, the model texture mapping module includes:

[0136] An environmental attribute determination unit is used to determine scene environmental attribute information in a target scene based on the image; wherein the scene environmental attribute information includes at least: seasonal information and weather information;

[0137] The texture mapping unit is used to generate a model texture based on the scene environment attribute information and map the model texture onto the initial three-dimensional model to obtain the reference three-dimensional model.

[0138] Optional, texture mapping unit, specifically used for:

[0139] Determine the device attribute parameters of the image acquisition device, and determine the perspective transformation relationship between the image and the initial three-dimensional model based on the device attribute parameters;

[0140] Based on the perspective transformation relationship, perform perspective transformation on the three-dimensional points in the second region of the initial three-dimensional model to obtain the corresponding texture coordinates of the second region;

[0141] The model textures corresponding to each texture coordinate are generated based on the scene environment attribute information.

[0142] Optionally, the device attribute parameters include at least device position information, rotation angle information, and intrinsic parameter matrix;

[0143] The texture mapping unit, including the perspective transformation matrix determination subunit, is used for:

[0144] The perspective transformation matrix is ​​determined based on the device position information, rotation angle information, and intrinsic parameter matrix, serving as the perspective transformation relationship between the image and the initial 3D model.

[0145] Optionally, the splicing model adjustment module includes a first adjustment unit, specifically used for:

[0146] The image is analyzed based on a pre-trained illumination estimation model to obtain scene illumination parameter information, which is then used as another scene attribute information. The scene illumination parameter information includes scene light source direction information, scene illumination intensity information, and scene ambient light color information.

[0147] The lighting parameters of the model in the initial stitching model are adjusted based on the scene lighting parameters.

[0148] Optionally, the splicing model adjustment module includes a second adjustment unit, specifically used for:

[0149] Other scene attribute information is extracted from the image, wherein the other scene attribute information includes at least one of the following: scene shadow information, scene reflection information, and scene highlight information;

[0150] Determine the corresponding model attribute information based on the other scene attribute information;

[0151] The other scene attribute information and the corresponding model attribute information are concatenated to obtain concatenated attribute information, and the initial concatenated model is adjusted according to the concatenated attribute information.

[0152] Optionally, the device also includes an illumination consistency optimization module, which is specifically used before obtaining the target stitching model:

[0153] The illumination intensity error is determined based on the illumination intensity of each pixel in the third region of the image and the corresponding region in the initial stitching model.

[0154] The optimization target is determined based on the light intensity error, and the light parameters in the initial stitching model are iteratively optimized.

[0155] Optional, splicing module, specifically used for:

[0156] The image stitching weight of the first pixel is determined based on the pixel value of the first pixel in the fourth region of the image.

[0157] The first stitching value of the first pixel is determined by the product of the pixel value of the first pixel in the image and the image stitching weight;

[0158] Determine the second pixel point in the reference 3D model that corresponds to the first pixel point;

[0159] The model stitching weight of the second pixel is determined based on the image stitching weight of the first pixel, and the second stitching value of the second pixel is determined based on the product of the pixel value of the second pixel in the reference 3D model and the model stitching weight.

[0160] Based on the sum of the first stitching value and the second stitching value, the pixel value of the pixel corresponding to the second pixel in the initial stitching model is determined.

[0161] The video stitching device in a three-dimensional scene provided in the embodiments of the present invention can execute the video stitching method in a three-dimensional scene provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0162] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations and do not violate public order and good morals.

[0163] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0164] Figure 6 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0165] like Figure 6As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0166] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0167] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as video stitching in a 3D scene.

[0168] In some embodiments, video stitching in the method's three-dimensional scene can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the video stitching in the method's three-dimensional scene described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform video stitching in the method's three-dimensional scene by any other suitable means (e.g., by means of firmware).

[0169] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific reference products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0170] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0171] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0172] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0173] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data servers), or computing systems that include switching components (e.g., application servers), or computing systems that include front-end components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such back-end, switching, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0174] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0175] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0176] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0177] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A video stitching method in a three-dimensional scene, characterized in that, The method includes: Acquire the initial 3D model of the target scene and the image of the first region in the target scene acquired by the image acquisition device; Based on the scene environment attribute information in the image, the initial 3D model is texture-mapped to obtain a reference 3D model; The reference 3D model and the image are stitched together to obtain an initial stitched model; Based on the image, other scene attribute information in the target scene is determined, and the initial stitching model is adjusted based on the other scene attribute information to obtain the target stitching model.

2. The method according to claim 1, characterized in that, Based on the scene environment attribute information in the image, texture mapping is performed on the initial 3D model to obtain a reference 3D model, including: The scene environment attribute information in the target scene is determined based on the image; wherein, the scene environment attribute information includes at least: seasonal information and weather information; The model texture is generated based on the scene environment attribute information, and the model texture is mapped onto the initial 3D model to obtain the reference 3D model.

3. The method according to claim 2, characterized in that, Generate model textures based on the scene environment attribute information, including: Determine the device attribute parameters of the image acquisition device, and determine the perspective transformation relationship between the image and the initial three-dimensional model based on the device attribute parameters; Based on the perspective transformation relationship, perform perspective transformation on the three-dimensional points in the second region of the initial three-dimensional model to obtain the corresponding texture coordinates of the second region; The model textures corresponding to each texture coordinate are generated based on the scene environment attribute information.

4. The method according to claim 3, characterized in that, in, The device attribute parameters include at least device position information, rotation angle information, and intrinsic parameter matrix; Determining the perspective transformation relationship between the image and the initial 3D model based on the device attribute parameters includes: The perspective transformation matrix is ​​determined based on the device position information, rotation angle information, and intrinsic parameter matrix, serving as the perspective transformation relationship between the image and the initial 3D model.

5. The method according to claim 1, characterized in that, Determine other scene attribute information in the target scene based on the image, and adjust the initial stitching model based on the other scene attribute information, including: The image is analyzed based on a pre-trained illumination estimation model to obtain scene illumination parameter information, which is then used as another scene attribute information. The scene illumination parameter information includes scene light source direction information, scene illumination intensity information, and scene ambient light color information. The lighting parameters of the model in the initial stitching model are adjusted based on the scene lighting parameters.

6. The method according to claim 1, characterized in that, Determine other scene attribute information in the target scene based on the image, and adjust the initial stitching model based on the other scene attribute information, including: Other scene attribute information is extracted from the image, wherein the other scene attribute information includes at least one of the following: scene shadow information, scene reflection information, and scene highlight information; Determine the corresponding model attribute information based on the other scene attribute information; The other scene attribute information and the corresponding model attribute information are concatenated to obtain concatenated attribute information, and the initial concatenated model is adjusted according to the concatenated attribute information.

7. The method according to claim 1, characterized in that, Before obtaining the target stitching model, the method further includes: The illumination intensity error is determined based on the illumination intensity of each pixel in the third region of the image and the corresponding region in the initial stitching model. The optimization target is determined based on the light intensity error, and the light parameters in the initial stitching model are iteratively optimized.

8. The method according to claim 1, characterized in that, The reference 3D model and the image are stitched together to obtain an initial stitched model, including: The image stitching weight of the first pixel is determined based on the pixel value of the first pixel in the fourth region of the image. The first stitching value of the first pixel is determined by the product of the pixel value of the first pixel in the image and the image stitching weight; Determine the second pixel point in the reference 3D model that corresponds to the first pixel point; The model stitching weight of the second pixel is determined based on the image stitching weight of the first pixel, and the second stitching value of the second pixel is determined based on the product of the pixel value of the second pixel in the reference 3D model and the model stitching weight. Based on the sum of the first stitching value and the second stitching value, the pixel value of the pixel corresponding to the second pixel in the initial stitching model is determined.

9. A video stitching device for a three-dimensional scene, characterized in that, The device includes: The model and image acquisition module is used to acquire the initial 3D model of the target scene and the image of the first region in the target scene acquired by the image acquisition device; The model texture mapping module is used to perform texture mapping on the initial 3D model based on the scene environment attribute information in the image to obtain a reference 3D model. The stitching module is used to stitch the reference 3D model and the image to obtain an initial stitched model; The stitching model adjustment module is used to determine other scene attribute information in the target scene based on the image, and adjust the initial stitching model based on the other scene attribute information to obtain the target stitching model.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which enables the at least one processor to perform the video stitching method in a three-dimensional scene according to any one of claims 1-8.

Images