A real-time shooting guidance system, method, device and medium

Abstract

The application belongs to the field of image processing and photography technology, and particularly relates to a real-time shooting guiding system, method, device and medium. The system comprises an image acquisition unit, an image processing unit and a guiding output unit. The image acquisition unit is used for shooting a target local scene from multiple different spatial points to obtain a multi-view image sequence and synchronous spatial pose parameters. The image processing unit fuses the multi-view image sequence and the synchronous spatial pose parameters according to a preset spatial coding rule to construct a high-dimensional scene model. The guiding output unit maps a current picture obtained by real-time shooting and a selected target reference picture to the high-dimensional scene model to generate visualized guiding information for adjusting shooting station or device parameters. The application solves the problem that the prior art cannot provide real-time, dynamic and visualized guidance in the shooting process.

Description

Technical Field

[0001] This invention belongs to the field of image processing and photography technology, and specifically relates to a real-time shooting guidance system, method, device and medium. Background Technology

[0002] The continuous evolution of digital imaging technology has driven the widespread adoption of photographic equipment, significantly enhancing its hardware performance and automation capabilities. However, for non-professional photographers, efficiently completing tasks such as composition design, angle selection, lighting adjustment, and capturing key moments during actual shooting remains a common challenge. Users often rely on limited personal experience, repeatedly trying and failing, gradually approaching the ideal effect through multiple previews and adjustments. This process not only consumes a lot of time and energy but also makes it difficult to consistently output high-quality images, resulting in low shooting efficiency and a low success rate.

[0003] Current photography assistance technologies primarily focus on post-capture image processing. For example, various professional image editing software offer post-processing adjustments such as exposure correction, color balance, and composition optimization; AI-based image generation tools can synthesize new content based on text descriptions; and photo management platforms emphasize image archiving, categorization, and retrieval. While these technologies have value in improving image quality or expanding content sources, they are essentially post-capture interventions on existing images, failing to intervene in the decision-making and execution process during shooting and thus unable to address the dynamic guidance needs during real-time shooting. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time shooting guidance system, method, device, and medium to solve the problem that existing technologies cannot provide real-time guidance for taking pictures.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a real-time shooting guidance system, comprising an image acquisition unit, an image processing unit, and a guidance output unit; The image acquisition unit is used to capture images of the same local scene from multiple different spatial points in the overall scene, so as to obtain a multi-view image sequence of the local scene and simultaneously acquire the corresponding spatial pose parameters at each capture. The image processing unit is used to fuse multi-view image sequences and corresponding spatial pose parameters according to preset spatial coding rules to construct a high-dimensional scene model; wherein, the high-dimensional scene model includes the relationship between the images of the target local scene captured at each spatial point and the corresponding spatial pose parameters. The guidance output unit is used to map the current image captured by the user in real time and the selected target reference image to a high-dimensional scene model. By calculating the difference between the spatial pose parameters corresponding to the current image and the target reference image in the high-dimensional scene model, it generates concrete guidance information for adjusting the shooting position or shooting equipment parameters.

[0006] The above solution addresses the problem that existing technologies can only provide post-processing or offline suggestions for single images, failing to offer real-time, dynamic, and concrete guidance during the shooting process. By constructing a high-dimensional scene model, the system can generate specific and actionable guidance instructions such as "how to move, how to rotate the lens, and how to zoom" based on the spatial and parameter differences between the real-time and target images. This transforms shooting guidance from abstract judgments of "composition quality" to intuitive "action execution" instructions, reducing the user's learning and trial-and-error costs. Composition and parameters can be optimized instantly during shooting, improving shooting efficiency and final image quality.

[0007] Furthermore, within the overall scene, images of the same local scene are captured from multiple different spatial points to obtain a multi-view image sequence of the local scene, and the spatial pose parameters corresponding to each capture are simultaneously acquired. Specifically, this includes the following steps: The target local scene is divided into spatial regions to obtain multiple sub-regions; The target local scene is captured from multiple preset spatial points; the image obtained from each spatial point covers a portion of the adjacent sub-regions of the target local scene. Record the spatial position of the imaging device, the shooting angle, and the shooting focal length at each shooting time, as spatial pose parameters.

[0008] The above-described scheme, utilizing a structured and gridded acquisition method, provides precisely aligned and spatially defined raw data for subsequent processing. Compared to unplanned, random shooting, this systematic data acquisition method ensures that the multi-view image sequence completely covers the target scene and that there are known overlaps and adjacencies between them. Simultaneously, the synchronously recorded pose parameters enable a dimensional upscaling from two-dimensional image information to three-dimensional spatial observation, which is an indispensable prerequisite for subsequent spatial difference calculations and path planning.

[0009] Furthermore, based on preset spatial coding rules, the multi-view image sequences and their corresponding spatial pose parameters are fused to construct a high-dimensional scene model, specifically including the following steps: In a multi-view image sequence, the images captured from each spatial point are decomposed according to the sub-regions covered by the images to obtain the pixel data corresponding to each sub-region. According to the preset data structure, the pixel data obtained from all spatial points are arranged and combined in sequence into a small region pixel array; wherein, the preset data structure defines the arrangement position and repetition pattern of the pixel data of each spatial point in the small region pixel array; Convolution operations are performed on small-area pixel arrays to fuse information from overlapping areas between different images, resulting in a set of non-overlapping images actually captured from each spatial point. A high-dimensional scene model is constructed based on a set of non-overlapping images and their corresponding spatial pose parameters.

[0010] The above scheme encodes spatial relationships into the structure of a data array through permutations and combinations of a pre-defined data structure; convolution operations separate images from different spatial points. This solves the problem of reconstructing a scene base image with clear spatial correspondences from raw, chaotic data. The constructed high-dimensional scene model is a multi-dimensional panoramic image that has removed viewpoint interference, is information-pure, and has precise pose labels.

[0011] Furthermore, the preset data structure is a data matrix; Based on the preset data structure, the pixel data obtained from all spatial points are sequentially arranged and combined into a small region pixel array, specifically including the following steps: Determine the row and column configuration of the data matrix, wherein the rows or columns of the data matrix are associated with one or more spatial points, and the elements of the data matrix are pixel data blocks covering multiple sub-regions obtained by decomposing the images captured from the corresponding spatial points. The pixel data blocks corresponding to each spatial point are filled into the specified positions of the data matrix according to the arrangement and repetition rules defined by the data structure. A small-area pixel array is generated based on the filled data matrix.

[0012] By defining the association between the rows and columns of a matrix and spatial points, and with each element being a pixel data block representing a specific point, spatial logic (which point, which sub-regions it covers) is mapped to matrix operations. This facilitates computer processing and computation, improving the efficiency and programmability of data integration. A fully filled data matrix becomes an intermediate state containing all the original information and its spatial relationships, providing a structurally sound input for the accurate execution of subsequent convolution operations.

[0013] Furthermore, convolution operations are performed on the small pixel array to fuse information from overlapping areas between different images, resulting in a set of non-overlapping images actually captured from each spatial point. This process includes the following steps: Design at least one convolution kernel, the parameters of which are configured to generate image data corresponding to a single spatial point when performing sliding convolution calculations on a small region of pixel arrays. The convolution kernel is applied to a small region pixel array to perform convolution calculation, and the convolution result is obtained. Based on the known distribution position of the sub-region associated with each spatial point in the small region pixel array, the convolution result is recombined into an intermediate image that corresponds one-to-one with each spatial point. Based on the intermediate images and the spatial pose parameters corresponding to each recorded spatial point, a set of non-overlapping images is reconstructed; each image corresponds uniquely to a spatial point.

[0014] The above solution solves the problem of information mixing caused by overlapping fields of view in multi-view images. The output is a clean scene image observed from each independent viewpoint, which enables a clear and accurate one-to-one correspondence between the image and spatial points in the high-dimensional scene model, ensuring the accuracy of subsequent difference calculations.

[0015] Furthermore, based on the non-overlapping set of images and the corresponding spatial pose parameters, a high-dimensional scene model is constructed, specifically including the following steps: Create a scene model data structure to store the image and its corresponding spatial pose parameters; Each frame in the set of non-overlapping frames is stored in the scene model data structure based on its unique corresponding spatial point. The spatial pose parameters corresponding to each frame, including spatial point information, shooting angle and shooting focal length, are associated and stored in the scene model data structure to establish an index relationship between the frame and the spatial pose parameters. Based on all established index relationships, a high-dimensional scene model is constructed; the high-dimensional scene model can query and locate the corresponding scene by using at least one parameter among spatial point location, shooting angle, or shooting focal length.

[0016] By establishing a bidirectional index relationship between the image and spatial pose parameters, this high-dimensional scene model essentially becomes a powerful spatial database. It supports multiple query modes, such as querying the corresponding image by spatial point location, or inferring the possible shooting pose by image features.

[0017] Furthermore, the current image captured by the user in real time and the selected target reference image are mapped into a high-dimensional scene model. By calculating the difference in spatial pose parameters between the current image and the target reference image in the high-dimensional scene model, concrete guidance information for adjusting the shooting position or shooting equipment parameters is generated. Specifically, this includes the following steps: In the high-dimensional scene model, the target spatial point, target shooting angle and target focal length corresponding to the current image and the target reference image are determined respectively; Calculate the spatial point difference between the spatial point corresponding to the current image and the target spatial point corresponding to the target reference image, calculate the angular offset between the shooting angle corresponding to the current image and the target shooting angle, and calculate the focal length change between the focal length corresponding to the current image and the target focal length. Based on spatial location differences, angular offsets, and focal length changes, concrete guidance information is generated, including specific movement directions, angular adjustments, and focal length adjustments.

[0018] By calculating spatial position differences, angular offsets, and focal length changes, the system can provide specific guidance. This guidance method completely changes the limitations of traditional photography aids that only provide abstract evaluations, allowing even non-professional users to gradually adjust to the optimal shooting position and parameters, just like following a navigation system. This enables them to reliably and repeatedly obtain high-quality shooting results, truly realizing the intelligentization and de-emphasization of the shooting process.

[0019] In a second aspect, the present invention provides a real-time shooting guidance method, comprising: In the overall scene, the same local scene of the target is captured from multiple different spatial points to obtain a multi-view image sequence of the local scene of the target, and the corresponding spatial pose parameters are collected at each capture. Based on the preset spatial coding rules, multi-view image sequences and their corresponding spatial pose parameters are fused to construct a high-dimensional scene model; the high-dimensional scene model includes the correlation between the images of the target local scene captured at each spatial point and the corresponding spatial pose parameters. The system maps the current image captured by the user in real time and the selected target reference image into a high-dimensional scene model. By calculating the difference in spatial pose parameters between the current image and the target reference image in the high-dimensional scene model, it generates concrete guidance information for adjusting the shooting position or shooting equipment parameters.

[0020] In a third aspect, the present invention provides an electronic device including a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the real-time shooting guidance method described above.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium storing at least one instruction that, when executed by a processor, implements the real-time shooting guidance method. Attached Figure Description

[0022] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a structural block diagram of a real-time shooting guidance system according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the construction of a high-dimensional scene model in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the generation of a reference photograph in an embodiment of the present invention; Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0024] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0025] Example 1 like Figure 1 As shown, this application provides a real-time shooting guidance system, including an image acquisition unit, an image processing unit, and a guidance output unit; The image acquisition unit is used to capture images of the same local scene from multiple different spatial points in the overall scene, so as to obtain a multi-view image sequence of the local scene and simultaneously acquire the corresponding spatial pose parameters at each capture. The image processing unit is used to fuse multi-view image sequences and corresponding spatial pose parameters according to preset spatial coding rules to construct a high-dimensional scene model; wherein, the high-dimensional scene model includes the relationship between the images of the target local scene captured at each spatial point and the corresponding spatial pose parameters. The guidance output unit is used to map the current image captured by the user in real time and the selected target reference image to a high-dimensional scene model. By calculating the difference between the spatial pose parameters corresponding to the current image and the target reference image in the high-dimensional scene model, it generates concrete guidance information for adjusting the shooting position or shooting equipment parameters.

[0026] In one embodiment, within the overall scene, images of the same local scene are captured from multiple different spatial points to obtain a multi-view image sequence of the local scene, and spatial pose parameters corresponding to each capture are simultaneously acquired. Specifically, this includes the following steps: The target local scene is divided into spatial regions to obtain multiple sub-regions; The target local scene is captured from multiple preset spatial points; the image obtained from each spatial point covers a portion of the adjacent sub-regions of the target local scene. Record the spatial position of the imaging device, the shooting angle, and the shooting focal length at each shooting time, as spatial pose parameters.

[0027] Specifically, spatial region division is used to discretize the target local scene, and the fineness of the division determines the spatial resolution of the final high-dimensional scene model. Shooting from multiple preset spatial points ensures that complete images of the target local scene can be captured from different perspectives, especially the spatial connections between sub-regions and the perspective and lighting differences caused by changes in viewpoint. Simultaneously recorded pose parameters (spatial point, shooting angle, shooting focal length) allow each set of pixel data to be traced back to its original observation geometry in subsequent processing.

[0028] As an example, suppose the target local scene is a landscape viewpoint. For example... Figure 2 As shown, the system can virtually divide the area into a grid of 3 rows and 5 columns, comprising 15 sub-regions, labeled A1 to C5 (A1, A2, A3, A4, A5, B1, B2, B3, B4, B5, C1, C2, C3, C4, C5). A1 to A5 form the first row, B1 to B5 the second row, and C1 to C5 the third row, used to indicate the shooting range. Four preset shooting spatial points are provided: Spatial Point 1 (capable of shooting A1, A2, A3, B1, B2, B3), Spatial Point 2 (capable of shooting A3, A4, A5, B3, B4, B5), Spatial Point 3 (capable of shooting B1, B2, B3, C1, C2, C3), and Spatial Point 4 (capable of shooting B3, B4, B5, C3, C4, C5).

[0029] A drone equipped with a gyroscope, accelerometer, and GPS / visual SLAM module can be used to sequentially capture images from these four spatial locations. At spatial location 1, images covering the sub-regions {A1, A2, A3, B1, B2, B3} are captured, and the GPS coordinates (or relative coordinates), attitude angles (pitch, yaw, roll), and focal length (e.g., equivalent to 28mm) are recorded. Similarly, images are captured at spatial locations 2, 3, and 4, covering {A3, A4, A5, B3, B4, B5}, {B1, B2, B3, C1, C2, C3}, and {B3, B4, B5, C3, C4, C5}, respectively. Ultimately, four images (a multi-view image sequence) and their corresponding four sets of pose parameters are obtained.

[0030] Alternatively, a device equipped with an imaging device, such as a mobile phone, camera, or mobile robot, that carries a gyroscope, accelerometer, or GPS can be used for shooting.

[0031] In one embodiment, a high-dimensional scene model is constructed by fusing multi-view image sequences and their corresponding spatial pose parameters according to a preset spatial coding rule, such as... Figure 2 As shown, the specific steps include the following: In a multi-view image sequence, the images captured from each spatial point are decomposed according to the sub-regions covered by the images to obtain the pixel data corresponding to each sub-region. According to the preset data structure, the pixel data obtained from all spatial points are arranged and combined in sequence into a small region pixel array; wherein, the preset data structure defines the arrangement position and repetition pattern of the pixel data of each spatial point in the small region pixel array; the preset data structure is a data matrix; Convolution operations are performed on small-area pixel arrays to fuse information from overlapping areas between different images, resulting in a set of non-overlapping images actually captured from each spatial point. A high-dimensional scene model is constructed based on a set of non-overlapping images and their corresponding spatial pose parameters.

[0032] A high-dimensional scene model can be understood as a multi-dimensional panoramic photograph.

[0033] In a more specific embodiment, the images captured from each spatial point in the multi-view image sequence are decomposed according to the sub-regions covered by the images to obtain the pixel data corresponding to each sub-region.

[0034] Specifically, the decomposition step involves dividing each image into smaller, regularized image blocks based on its known coverage area. Each image block corresponds to all pixel information of a sub-region. In essence, the decomposition step discretizes continuous target scene images into data units aligned with a spatial grid. Through decomposition, observation data from different spatial points concerning the same sub-region can be separated. Unlike traditional feature extraction, this decomposition step fully preserves the original pixel information, ensuring the information fidelity of the reconstructed high-dimensional scene model.

[0035] As an example, the image captured at spatial point 1 is processed. It is known that this image covers sub-regions {A1, A2, A3, B1, B2, B3}. The system divides this image into 6 independent image blocks based on the preset sub-regions. The image block corresponding to sub-region A1 contains all the pixel data of that sub-region and is denoted as pixel data block A11 (representing the pixel data block from sub-region A1 captured by spatial point 1). Similarly, A21 (representing the pixel data block from sub-region A2 captured by spatial point 1), A31 (representing the pixel data block from sub-region A3 captured by spatial point 1), B11 (representing the pixel data block from sub-region B1 captured by spatial point 1), B21 (representing the pixel data block from sub-region B2 captured by spatial point 1), and B31 (representing the pixel data block from sub-region B3 captured by spatial point 1) are obtained.

[0036] The image at spatial point 2 is decomposed into A32 (representing the pixel data block of sub-region A3 captured by spatial point 2), A42 (representing the pixel data block of sub-region A4 captured by spatial point 2), A52 (representing the pixel data block of sub-region A5 captured by spatial point 2), B32 (representing the pixel data block of sub-region B3 captured by spatial point 2), B42 (representing the pixel data block of sub-region B4 captured by spatial point 2), and B52 (representing the pixel data block of sub-region B5 captured by spatial point 2).

[0037] The image at spatial point 3 is decomposed into B13 (representing pixel data blocks from sub-region B1 captured by spatial point 3), B23 (representing pixel data blocks from sub-region B2 captured by spatial point 3), B33 (representing pixel data blocks from sub-region B3 captured by spatial point 3), C13 (representing pixel data blocks from sub-region C1 captured by spatial point 3), C23 (representing pixel data blocks from sub-region C2 captured by spatial point 3), and C33 (representing pixel data blocks from sub-region C3 captured by spatial point 3).

[0038] The image at spatial point 4 is decomposed into B34 (representing pixel data blocks from sub-region B3 captured by spatial point 4), B44 (representing pixel data blocks from sub-region B4 captured by spatial point 4), B54 (representing pixel data blocks from sub-region B5 captured by spatial point 4), C34 (representing pixel data blocks from sub-region C3 captured by spatial point 4), C44 (representing pixel data blocks from sub-region C4 captured by spatial point 4), and C54 (representing pixel data blocks from sub-region C5 captured by spatial point 4).

[0039] It should be noted that sub-regions A3, C3, B1~B5, etc., were observed by multiple spatial points, resulting in multiple sets of pixel data (for example, A31 and A32 were actually taken from spatial point 1 and spatial point 2, respectively, of the same sub-region A3). The pixel data of different spatial points corresponding to the same sub-region differ due to different viewing angles and lighting conditions.

[0040] In a more specific embodiment, according to a preset data structure, the pixel data obtained from the decomposition of all spatial points are sequentially arranged and combined into a small region pixel array, specifically including the following steps: Determine the row and column configuration of the data matrix, wherein the rows or columns of the data matrix are associated with one or more spatial points, and the elements of the data matrix are pixel data blocks covering multiple sub-regions obtained by decomposing the images captured from the corresponding spatial points. The pixel data blocks corresponding to each spatial point are filled into the specified positions of the data matrix according to the arrangement and repetition rules defined by the data structure. A small-area pixel array is generated based on the filled data matrix.

[0041] Specifically, the data matrix serves as a template, and its row and column configuration and element definitions constitute a spatial encoding protocol. Filling the data matrix with scattered pixel data blocks essentially maps the observation data of three-dimensional space (spatial points, sub-regions) onto a two-dimensional, extended data plane (i.e., a small-area pixel array) in a way that strengthens spatial relationships and point correlations. This explicitly encodes information such as "which data come from the same point" and "which sub-regions are jointly observed by which points" at the data level. This allows subsequent convolutional neural networks to more easily learn the differences and correlations between points, thereby effectively separating overlapping information.

[0042] As an example, the data matrix is ​​defined as a 4x6 matrix.

[0043] The specific rules are as follows: All pixel data blocks from a single spatial point that cover six sub-regions are considered as a single unit, called a combined block. For example, the combined block K1 for spatial point 1 is [A11, A21, A31; B11, B21, B31] (a 2x3 image block matrix). Similarly, K2 (spatial point 2), K3 (spatial point 3), and K4 (spatial point 4). The filling pattern of the data matrix is ​​shown in Table 1 below: Table 1 Data Matrix

[0044] After filling the data matrix with the specific pixel values ​​of K1, K2, K3, and K4 according to this pattern, all elements of the entire data matrix (each element itself is a small matrix) are expanded in row-major order, ultimately resulting in a large two-dimensional pixel array with a highly repetitive and regular structure, namely, a small-area pixel array. This small-area pixel array contains multiple expressions of the original image data and its spatial relationships.

[0045] The small-area pixel array is shown in Table 2 below: Table 2 Small Region Pixel Array

[0046] In a more specific embodiment, convolution operations are performed on a small region pixel array to fuse information from overlapping areas between different images, resulting in a set of non-overlapping images actually captured from each spatial point. This includes the following steps: Design at least one convolution kernel, the parameters of which are configured to generate image data corresponding to a single spatial point when performing sliding convolution calculations on a small region of pixel arrays. The convolution kernel is applied to a small region pixel array to perform convolution calculation, and the convolution result is obtained. Based on the known distribution position of the sub-region associated with each spatial point in the small region pixel array, the convolution result is recombined into an intermediate image that corresponds one-to-one with each spatial point. Based on the intermediate images and the spatial pose parameters corresponding to each recorded spatial point, a set of non-overlapping images is reconstructed; each image corresponds uniquely to a spatial point.

[0047] Specifically, since the small-region pixel array is constructed according to a preset pattern, information from the same spatial point appears periodically within the small-region pixel array, while information from different spatial points is interleaved. Through training, the convolutional kernel can learn this periodic pattern. During forward propagation, the sliding calculation of the convolutional kernel can resonate and enhance the information from the same periodically appearing point, while suppressing non-periodic interference information from other points, thereby achieving the effect of information fusion and purification. The convolution result is a set of feature maps. Based on the known mapping relationship between spatial points and sub-regions during encoding, these feature maps can be recombined into several images, each image fusing purified images from the corresponding spatial point. Finally, combined with the original pose parameters of the spatial point (such as shooting focal length), geometric and optical reverse corrections are performed on the intermediate images to reconstruct a non-overlapping image that conforms to the original shooting geometry of the spatial point.

[0048] As an example, a set of convolutional kernels can be designed for small pixel arrays. For instance, the weights of a convolutional kernel are trained such that when it slides into a region in the small pixel array that matches the pattern of K1, it generates high activation; when it slides into regions where K2, K3, and K4 appear, the activation is suppressed. The convolutional results (feature maps) are analyzed, and according to a pre-defined mapping table (e.g., the activation peak position of the i-th channel of the feature map corresponds to the j-th sub-region of the K1 combination block of spatial point 1 in the original array), the feature values ​​of different channels are redistributed into containers of 15 sub-regions, forming four preliminary scene layers, corresponding to spatial points 1, 2, 3, and 4, respectively. Subsequently, for each spatial point, using its recorded shooting focal length, the scene layer of that spatial point is subjected to anti-distortion and viewpoint normalization processing; using its shooting angle, the image is appropriately rotated and corrected, ultimately generating four clear images with correct field of view and non-overlapping content.

[0049] In a more specific embodiment, a high-dimensional scene model is constructed based on a set of non-overlapping images and their corresponding spatial pose parameters, specifically including the following steps: Create a scene model data structure to store the image and its corresponding spatial pose parameters; Each frame in the set of non-overlapping frames is stored in the scene model data structure based on its unique corresponding spatial point. The spatial pose parameters corresponding to each frame, including spatial point information, shooting angle and shooting focal length, are associated and stored in the scene model data structure to establish an index relationship between the frame and the spatial pose parameters. Based on all established index relationships, a high-dimensional scene model is constructed; the high-dimensional scene model can query and locate the corresponding scene by using at least one parameter among spatial point location, shooting angle, or shooting focal length.

[0050] Specifically, the scene model data structure can be a graph-structured database containing nodes and edges. Each node in the graph database represents an observation state and contains two main fields: image and spatial pose parameters (spatial point location, shooting angle, and shooting focal length). An index relationship is established between the image and the spatial pose parameters, serving as a key for fast lookups within the scene model data structure. For example, the ID of the spatial point can be used as the primary key, while secondary indexes can be created for the shooting angle and shooting focal length. After the high-dimensional scene model is constructed, it supports both primary key and secondary index queries; for example, inputting a spatial point ID returns the corresponding image and complete pose.

[0051] In one embodiment, the current image captured by the user in real time and the selected target reference image are mapped into a high-dimensional scene model. By calculating the difference in spatial pose parameters between the current image and the target reference image in the high-dimensional scene model, concrete guidance information for adjusting the shooting position or shooting equipment parameters is generated. Specifically, the steps include the following: In the high-dimensional scene model, the target spatial point, target shooting angle and target focal length corresponding to the current image and the target reference image are determined respectively; Calculate the spatial point difference between the spatial point corresponding to the current image and the target spatial point corresponding to the target reference image, calculate the angular offset between the shooting angle corresponding to the current image and the target shooting angle, and calculate the focal length change between the focal length corresponding to the current image and the target focal length. Based on spatial location differences, angular offsets, and focal length changes, concrete guidance information is generated, including specific movement directions, angular adjustments, and focal length adjustments.

[0052] In one embodiment, an AI model is used to generate at least one target reference image based on the user-input shooting requirements and a pre-constructed high-dimensional scene model, such as... Figure 3 As shown, the specific steps include the following: First, the user's shooting requirements are obtained. The user inputs their shooting preferences, style, or specific compositional intentions via voice and / or text. This shooting requirement information is processed into a text token sequence. Simultaneously, the constructed high-dimensional scene model (containing images and spatial pose parameters of multiple spatial points) is expressed as a visual token sequence. For example, each token in the visual token sequence can be an image patch from the corresponding spatial point, shooting angle, and associated pose code.

[0053] Next, the text token sequence and visual token sequence are fused using a pre-trained multimodal encoder. The multimodal encoder consists of components such as a digital transformation matrix, an information interaction matrix (like the attention layer in a Transformer), and a multilayer perceptron. Its operating mechanism is as follows: the digital transformation matrix maps the text and visual tokens to the same multidimensional semantic space, forming a multidimensional information matrix. This multidimensional information matrix includes both text feature matrices and visual feature matrices. In this multidimensional semantic space, different dimensions of the vectors are learned to correspond to different visual concepts (such as sky, texture, wide-angle, etc.).

[0054] Then, the correlation between text features and visual features is calculated through the information interaction matrix (which can employ a cross-attention mechanism), enabling text requirements to adaptively focus on and enhance the relevant visual parts in the high-dimensional scene model, and outputting a fused feature matrix.

[0055] Subsequently, a multilayer perceptron is used to perform nonlinear transformation and filtering on the fused feature matrix. The parameters of the multilayer perceptron are optimized during training to "score" and "purify" the fused information based on multiple evaluation criteria such as aesthetics and composition, thereby generating a latent code matrix.

[0056] Finally, a decoder is used to upsample the latent code matrix and convert it into image data through a digital inverse conversion matrix, generating one or more reference images that meet the user's needs and are based on the current real scene, allowing the user to select the target reference image.

[0057] In one embodiment, the AI ​​model is trained as follows: Specifically, the training of this AI model is a data-driven, end-to-end supervised learning process. Its goal is to enable the AI ​​model to generate reference photos that meet aesthetic standards and user needs based on input shooting requirements and high-dimensional scene models. The training process is described below: 1. Construction of training data triples: Each training sample consists of a data triple.

[0058] Input 1: High-dimensional scene model. Select a target local scene from the overall scene, capture the target local scene from multiple spatial point systems, and construct a high-dimensional scene model of the overall scene.

[0059] Input 2: Shooting Requirements Information. This consists of high-quality photos manually selected from the overall scene, accompanied by text describing their style or intent. This text will be processed into a sequence of text tokens.

[0060] Supervision Target: High-quality target images. These are high-quality photographs taken by professional photographers and selected through an objective evaluation system, taken within the same scene and using preset spatial points and pose parameters. These high-quality photographs serve as the reference images that the AI ​​model needs to learn and generate.

[0061] 2. Forward propagation and generation process: The high-dimensional scene model in the training samples is represented as a visual token sequence, which is then input into the model to be trained along with the text token sequence.

[0062] The model to be trained maps the two types of tokens to a multi-dimensional semantic space through its digital transformation matrix, forming the initial text feature matrix and visual feature matrix.

[0063] The correlation between the two is calculated using the information interaction matrix (emphasizing the cross-attention mechanism), and the fused feature matrix is ​​output.

[0064] The multilayer perceptron processes the fused feature matrix to generate a latent code matrix.

[0065] Finally, the decoder (whose core is the digital inverse transformation matrix) converts the latent code matrix into the generated image.

[0066] 3. Loss Calculation and Parameter Update: The difference between the image generated by the model under training and the high-quality target image of the supervised object is calculated to form a loss function. This loss function can combine pixel-level reconstruction loss and perceptual loss to ensure content and style similarity.

[0067] The backpropagation algorithm is used to update the weights and bias parameters of all these components by passing the loss gradient from the decoder output through the inverse digital transformation matrix, the multilayer perceptron, the information interaction matrix, and back to the frontmost digital transformation matrix.

[0068] This process iterates repeatedly, gradually giving the vector directions in the multi-dimensional semantic space within the model to be trained clear visual conceptual meanings. It also enables the information interaction matrix and the multilayer perceptron to select, combine, and optimize visual content that meets aesthetic standards from scene information according to textual requirements.

[0069] 4. External evaluation system: To ensure the quality and objectivity of the training set (high-quality target images), a quantitative evaluation system based on fuzzy hierarchical analysis was used for screening. This system defines evaluation dimensions and weights such as composition and lighting, and scores candidate photos. This quantitative evaluation system is only used to construct high-quality training data; its rules and weights are not directly encoded within the AI ​​model. All aesthetic judgment capabilities of the model are acquired implicitly from the data through the aforementioned end-to-end training.

[0070] As an example, the quantitative evaluation system is shown in Table 3 below: Table 3 Quantitative Evaluation System

[0071] Based on the relative importance of each evaluation dimension, fuzzy numbers are used to represent it. Commonly used fuzzy numbers include "1" (both are equally important), "3" (one is more important than the other), and "5" (one is significantly more important than the other). The relative importance of the evaluation dimensions is shown in Table 4. Table 4. Relative Importance of Evaluation Dimensions

[0072] In one embodiment, the spatial point difference between the spatial point corresponding to the current frame and the target spatial point corresponding to the target reference frame is calculated, the angular offset between the shooting angle corresponding to the current frame and the target shooting angle is calculated, and the focal length change between the focal length corresponding to the current frame and the target focal length is calculated, including: Since the target reference image is selected by the system from the generated reference images according to user needs, the reference image is bound to one or more optimal spatial points, shooting angles, and shooting focal length parameters in the high-dimensional scene model during generation. Therefore, its target spatial point P_target, target shooting angle A_target (which can include the three Euler angles of pitch, yaw, and roll), and target focal length F_target can be directly read from the high-dimensional scene model.

[0073] For the current image, the system performs real-time feature extraction and quickly matches it with the visual features of each node stored in the high-dimensional scene model to find the best-matching model node, initially determining a reference spatial point P_ref and a reference shooting angle A_ref. Simultaneously, the system obtains the device's current coarse spatial point P_sensor, shooting angle A_sensor, and shooting focal length F_sensor from the imaging device's sensors in real time. Then, using Kalman filtering algorithms, the visually matched pose (P_ref, A_ref) is fused with the sensor-based pose (P_sensor, A_sensor, F_sensor) to obtain a precise and smooth current spatial point P_current, current shooting angle A_current, and current focal length F_current.

[0074] Spatial point difference ΔP calculation: Treat the target spatial point and the current spatial point as two coordinate points in three-dimensional space. Calculate their difference vector: ΔP = P_target - P_current = [ΔX, ΔY, ΔZ]. This difference vector ΔP directly represents the three-dimensional translation required to move from the current position to the target position. The magnitude represents the distance moved, and the direction represents the direction of movement.

[0075] Calculation of angular offset ΔA: Calculate the difference between the three Euler angle components separately. ΔA = A_target - A_current = [ΔPitch (pitch difference), ΔYaw (yaw difference), ΔRoll (roll difference)]. Each angle component is calculated independently, in degrees or radians, with the sign indicating the direction of rotation (e.g., a positive ΔPitch indicates an upward pitch, and a negative ΔPitch indicates a downward pitch).

[0076] Focal length change ΔF calculation: Calculate the scalar difference in focal length: ΔF = F_target - F_current. The unit is millimeters or equivalent focal length value. The positive or negative sign indicates whether to zoom in (if ΔF > 0) or zoom out (if ΔF < 0).

[0077] In one embodiment, based on spatial location differences, angular offsets, and focal length changes, concrete guidance information is generated, including specific movement directions, angular adjustment amounts, and focal length adjustment values, as follows: Spatial movement guidance generation: This process converts the 3D spatial positional differences ΔP into user-understandable movement commands. First, ΔP is decomposed into the user's real-world coordinate system (e.g., forward-backward, left-right, up-down). For example, ΔP = [2.1, -0.5, 0.3] meters can be interpreted as "move forward 2.1 meters, move left 0.5 meters, and rise slightly by 0.3 meters." Simultaneously, in the device's augmented reality preview interface, the system renders a 3D arrow or path guide line in real-time, pointing from the current screen center towards the target direction. Its length and direction dynamically correspond to the magnitude and direction of ΔP.

[0078] Angle adjustment guidance generation: The angle offset ΔA is converted into device attitude adjustment instructions. For each Euler angle component, a clear prompt is generated. For example, ΔA=[5°,-10°,0°] is converted to: "Raise the device upwards by 5 degrees" and "Rotate the device to the right by 10 degrees." On the AR interface, a virtual level and angle scale are simultaneously displayed, visually showing the deviation between the current angle and the target angle, and dynamically indicating the adjustment direction until it reaches zero.

[0079] Focal length adjustment guidance generation: Converts the focal length change value ΔF into lens operation commands. Commands such as "Adjust the focal length to 35mm" or "Zoom in by 2mm". On the interface, a simulated focal length scale bar or slider will highlight the target focal length value and prompt the user to perform the corresponding operation.

[0080] The system integrates the parsed text instructions and graphic overlay elements (arrows, levels, scale bars) to form concrete guidance information. This concrete guidance information is output to the user in real time through at least one of the following methods: AR guidance graphics and concise text prompts are directly overlaid on the viewfinder or screen preview of the shooting device.

[0081] Using speech synthesis technology, the direction of movement, the amount of angle adjustment, and the value of focus adjustment are announced in real time in conversational language (e.g., "Please move slowly to the left front... Okay, stop. Now please raise your phone up a little...").

[0082] If the controlled object is a device with automatic adjustment capabilities (such as a gimbal camera or a drone), the system directly converts ΔP, ΔA, and ΔF into low-level control commands (such as PWM control signals for gimbal motors or flight control commands for drones), driving the device to automatically perform movement, turning, and zoom operations until the matching degree between the real-time image and the target reference image reaches a preset threshold.

[0083] Example 2 Based on the same inventive concept as the above embodiments, the present invention also provides a real-time shooting guidance method, including: In the overall scene, the same local scene of the target is captured from multiple different spatial points to obtain a multi-view image sequence of the local scene of the target, and the corresponding spatial pose parameters are collected at each capture. Based on the preset spatial coding rules, multi-view image sequences and their corresponding spatial pose parameters are fused to construct a high-dimensional scene model; the high-dimensional scene model includes the correlation between the images of the target local scene captured at each spatial point and the corresponding spatial pose parameters. The system maps the current image captured by the user in real time and the selected target reference image into a high-dimensional scene model. By calculating the difference in spatial pose parameters between the current image and the target reference image in the high-dimensional scene model, it generates concrete guidance information for adjusting the shooting position or shooting equipment parameters.

[0084] Example 3 like Figure 4 As shown, the present invention also provides an electronic device 100 for implementing a real-time shooting guidance method; The electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on at least one processor 102, and at least one communication bus 104.

[0085] The memory 101 can be used to store computer program 103. The processor 102 implements the steps of a real-time shooting guidance method of Embodiment 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0086] The memory 101 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0087] At least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 102 may be a microprocessor or any conventional processor. Processor 102 is the control center of electronic device 100, connecting various parts of electronic device 100 via various interfaces and lines.

[0088] The memory 101 in the electronic device 100 stores multiple instructions to implement a real-time shooting guidance method, and the processor 102 can execute multiple instructions to achieve the following: In the overall scene, the same local scene of the target is captured from multiple different spatial points to obtain a multi-view image sequence of the local scene of the target, and the corresponding spatial pose parameters are collected at each capture. Based on the preset spatial coding rules, multi-view image sequences and their corresponding spatial pose parameters are fused to construct a high-dimensional scene model; the high-dimensional scene model includes the correlation between the images of the target local scene captured at each spatial point and the corresponding spatial pose parameters. The system maps the current image captured by the user in real time and the selected target reference image into a high-dimensional scene model. By calculating the difference in spatial pose parameters between the current image and the target reference image in the high-dimensional scene model, it generates concrete guidance information for adjusting the shooting position or shooting equipment parameters.

[0089] Example 4 If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0090] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A real-time shooting guidance system, characterized in that, It includes an image acquisition unit, an image processing unit, and a guide output unit; The image acquisition unit is used to capture images of the same local scene from multiple different spatial points in the overall scene, so as to obtain a multi-view image sequence of the local scene and simultaneously acquire the corresponding spatial pose parameters at each capture. The image processing unit is used to fuse multi-view image sequences and corresponding spatial pose parameters according to preset spatial coding rules to construct a high-dimensional scene model; wherein, the high-dimensional scene model includes the relationship between the images of the target local scene captured at each spatial point and the corresponding spatial pose parameters. The guidance output unit is used to map the current image captured by the user in real time and the selected target reference image to a high-dimensional scene model. By calculating the difference between the spatial pose parameters corresponding to the current image and the target reference image in the high-dimensional scene model, it generates concrete guidance information for adjusting the shooting position or shooting equipment parameters.

2. The real-time shooting guidance system according to claim 1, characterized in that, In the overall scene, images of the same local scene are captured from multiple different spatial points to obtain a multi-view image sequence of the local scene, and the spatial pose parameters corresponding to each capture are collected simultaneously. The specific steps include the following: The target local scene is divided into spatial regions to obtain multiple sub-regions; The target local scene is captured from multiple preset spatial points; the image obtained from each spatial point covers a portion of the adjacent sub-regions of the target local scene. Record the spatial position of the imaging device, the shooting angle, and the shooting focal length at each shooting time, as spatial pose parameters.

3. The real-time shooting guidance system according to claim 2, characterized in that, Based on preset spatial coding rules, multi-view image sequences and their corresponding spatial pose parameters are fused to construct a high-dimensional scene model, specifically including the following steps: In a multi-view image sequence, the images captured from each spatial point are decomposed according to the sub-regions covered by the images to obtain the pixel data corresponding to each sub-region. According to the preset data structure, the pixel data obtained from all spatial points are arranged and combined in sequence into a small region pixel array; wherein, the preset data structure defines the arrangement position and repetition pattern of the pixel data of each spatial point in the small region pixel array; Convolution operations are performed on small-area pixel arrays to fuse information from overlapping areas between different images, resulting in a set of non-overlapping images actually captured from each spatial point. A high-dimensional scene model is constructed based on a set of non-overlapping images and their corresponding spatial pose parameters.

4. The real-time shooting guidance system according to claim 3, characterized in that, The default data structure is a data matrix; Based on the preset data structure, the pixel data obtained from all spatial points are sequentially arranged and combined into a small region pixel array, specifically including the following steps: Determine the row and column configuration of the data matrix, wherein the rows or columns of the data matrix are associated with one or more spatial points, and the elements of the data matrix are pixel data blocks covering multiple sub-regions obtained by decomposing the images captured from the corresponding spatial points. The pixel data blocks corresponding to each spatial point are filled into the specified positions of the data matrix according to the arrangement and repetition rules defined by the data structure. A small-area pixel array is generated based on the filled data matrix.

5. The real-time shooting guidance system according to claim 4, characterized in that, Convolution operations are performed on a small pixel array to fuse information from overlapping areas between different images, resulting in a set of non-overlapping images actually captured from each spatial point. The specific steps include: Design at least one convolution kernel, the parameters of which are configured to generate image data corresponding to a single spatial point when performing sliding convolution calculations on a small region of pixel arrays. The convolution kernel is applied to a small region pixel array to perform convolution calculation, and the convolution result is obtained. Based on the known distribution position of the sub-region associated with each spatial point in the small region pixel array, the convolution result is recombined into an intermediate image that corresponds one-to-one with each spatial point. Based on the intermediate images and the spatial pose parameters corresponding to each recorded spatial point, a set of non-overlapping images is reconstructed; each image corresponds uniquely to a spatial point.

6. The real-time shooting guidance system according to claim 5, characterized in that, Based on a set of non-overlapping images and their corresponding spatial pose parameters, a high-dimensional scene model is constructed, which includes the following steps: Create a scene model data structure to store the image and its corresponding spatial pose parameters; Each frame in the set of non-overlapping frames is stored in the scene model data structure based on its unique corresponding spatial point. The spatial pose parameters corresponding to each frame, including spatial point information, shooting angle and shooting focal length, are associated and stored in the scene model data structure to establish an index relationship between the frame and the spatial pose parameters. Based on all established index relationships, a high-dimensional scene model is constructed; the high-dimensional scene model can query and locate the corresponding scene by using at least one parameter among spatial point location, shooting angle, or shooting focal length.

7. The real-time shooting guidance system according to claim 6, characterized in that, The user's real-time captured image and the selected target reference image are mapped into a high-dimensional scene model. By calculating the difference in spatial pose parameters between the current image and the target reference image in the high-dimensional scene model, concrete guidance information is generated for adjusting the shooting position or shooting equipment parameters. The specific steps include the following: In the high-dimensional scene model, the target spatial point, target shooting angle and target focal length corresponding to the current image and the target reference image are determined respectively; Calculate the spatial point difference between the spatial point corresponding to the current image and the target spatial point corresponding to the target reference image, calculate the angular offset between the shooting angle corresponding to the current image and the target shooting angle, and calculate the focal length change between the focal length corresponding to the current image and the target focal length. Based on spatial location differences, angular offsets, and focal length changes, concrete guidance information is generated, including specific movement directions, angular adjustments, and focal length adjustments.

8. A real-time shooting guidance method, characterized in that, include: In the overall scene, the same local scene of the target is captured from multiple different spatial points to obtain a multi-view image sequence of the local scene of the target, and the corresponding spatial pose parameters are collected at each capture. Based on the preset spatial coding rules, multi-view image sequences and their corresponding spatial pose parameters are fused to construct a high-dimensional scene model; the high-dimensional scene model includes the relationship between the images of the target local scene captured at each spatial point and the corresponding spatial pose parameters. The system maps the current image captured by the user in real time and the selected target reference image into a high-dimensional scene model. By calculating the difference in spatial pose parameters between the current image and the target reference image in the high-dimensional scene model, it generates concrete guidance information for adjusting the shooting position or shooting equipment parameters.

9. An electronic device, characterized in that, It includes a processor and a memory, the processor being used to execute a computer program stored in the memory to implement the real-time shooting guidance method as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the real-time shooting guidance method as described in claim 8.

Images