Photography auxiliary method, system and equipment based on artificial intelligence and medium

Abstract

The invention is suitable for the technical field of photography, and provides an artificial intelligence-based photography auxiliary method, which comprises the following steps of: capturing a view-finding video stream in real time on a photographing view-finding interface of user equipment, and synchronously obtaining multi-modal data through an equipment sensor of the user equipment; based on the view-finding video stream, performing scene recognition and main body extraction by adopting an image recognition model to obtain a preliminary composition element; generating at least one candidate composition scheme by adopting a user intention prediction model on the basis of the preliminary composition elements in combination with the multi-modal data set; based on a pre-trained aesthetics evaluation model, performing aesthetics quality scoring on the plurality of candidate composition schemes, and selecting a recommended composition scheme; on the photographing view-finding interface, on the basis of the recommended composition scheme, the composition guide line is generated, and the voice guidance information is output for guiding the user to photograph, the auxiliary opportunity is advanced from'post-photographing 'to'in-photographing', each composition recommendation scheme has scene correlation and professional level, and personalized requirements of the user are met.

Description

Technical Field

[0001] This application belongs to the field of photography technology, and in particular relates to a photography assistance method, system, device and medium based on artificial intelligence. Background Technology

[0002] In modern photography, providing effective shooting techniques and composition advice is an important issue. Traditional photography teaching is usually conducted through courses, books, or offline instruction. However, these methods have obvious limitations, which has led to the emergence of AI-assisted photography.

[0003] Existing AI photography assistance solutions typically focus on post-processing analysis and commentary on already taken photos. For example, they may provide text suggestions such as "use the rule of thirds" after recognizing the content of the photo. However, such analysis and commentary are difficult to provide real-time and customized suggestions based on factors such as the specific scene (such as lighting and scenery), the subject (such as people, landscapes, and still life), and the equipment used by the photographer. They are reactive and lack real-time performance, resulting in a poor user experience and failing to meet the needs of users who want to quickly improve the quality of their photos in specific scenarios.

[0004] Therefore, there is an urgent need for a photography assistance solution that can provide real-time and accurate composition suggestions. Summary of the Invention

[0005] This application provides an artificial intelligence-based photography assistance method, system, device, and medium, which can solve one of the problems of the prior art mentioned above.

[0006] In a first aspect, embodiments of this application provide an artificial intelligence-based photography assistance method, comprising: On the user device's camera viewfinder, capture the viewfinder video stream in real time and simultaneously obtain multimodal data through the user device's sensors; Based on the framing video stream, an image recognition model is used to perform scene recognition and subject extraction to obtain preliminary composition elements; Based on the preliminary mapping elements and combined with the multimodal dataset, at least one candidate mapping scheme is generated using a user intent prediction model. Based on a pre-trained aesthetic evaluation model, the aesthetic quality of multiple candidate composition schemes is scored, and a recommended composition scheme is selected. In the photo framing interface, based on the recommended composition scheme, composition guide lines are generated and voice guidance information is output to guide the user in taking photos.

[0007] Furthermore, the step of capturing a viewfinder video stream in real time on the user device's camera viewfinder interface and simultaneously obtaining multimodal data through the user device's sensors includes: The viewfinder video stream is captured in real time using the user's device camera and preset video display attributes; Based on the inertial measurement unit of the user device, the system continuously acquires raw motion data of the user device. By analyzing the raw motion data, the user's holding posture and the motion state of the shot are determined. Based on the ambient light sensor and color temperature sensor of the user device, the ambient lighting conditions are collected, and the lighting conditions are combined with the viewfinder video stream for analysis to determine the lighting direction and light quality of the shooting scene.

[0008] Furthermore, based on the framing video stream, an image recognition model is used to perform scene recognition and subject extraction to obtain preliminary composition elements, including: Each frame of the viewfinder video stream is preprocessed to obtain a standardized image frame; The standardized image frames are input into a pre-trained image recognition model, and the image recognition model is used to perform scene classification and semantic segmentation to obtain scene classification results and semantic segmentation maps. Based on the scene classification results, dynamic subject filtering is performed, and the main visual subjects are identified and located by combining the semantic segmentation map. The scene classification results, semantic segmentation map, and main visual subjects are associated and encapsulated to form preliminary composition elements.

[0009] Furthermore, the step of generating at least one candidate mapping scheme based on the preliminary mapping elements and the multimodal data using a user intent prediction model includes: The initial mapping elements are structurally associated with the multimodal data to construct a multimodal intent understanding map; The multimodal intent understanding graph is input into the user intent prediction model. The user intent prediction result is obtained through the user intent prediction model and retrieved and reasoned in the graph composition scheme knowledge graph to generate multiple candidate graph composition schemes. For each candidate graphing scheme, parameter instantiation is performed in conjunction with the multimodal intent understanding graph to generate parameterized information.

[0010] Furthermore, the pre-trained aesthetic evaluation model scores the aesthetic quality of multiple candidate composition schemes and selects a recommended composition scheme, including: Based on the composition rules of each candidate composition scheme, image simulation is performed on the original viewfinder frame to generate a composition reference image of the corresponding candidate composition scheme. The composition reference image is input into a pre-trained aesthetic evaluation model, and the aesthetic evaluation model outputs an overall aesthetic score and a multi-dimensional aesthetic feature vector. The multidimensional aesthetic feature vectors are dynamically matched with the user intent prediction results, and the intent conformity weighted score of each candidate solution is calculated. By combining the overall aesthetic score and the intention conformity weighted score, a final comprehensive score is obtained for each candidate composition scheme, and a recommended composition scheme is determined based on the final comprehensive score.

[0011] Furthermore, in the photo-taking interface, based on the recommended composition scheme, a composition guide line is generated and voice guidance information is output to guide the user in taking photos, including: Based on the parametric information corresponding to the recommended composition scheme, dynamic composition guide lines are generated and rendered in the photo framing interface. The composition guide lines include static frame lines and dynamic semantic lines, wherein the static frame lines are used to represent the basic structure of the composition, and the dynamic semantic lines are used to highlight specific visual elements in the scene. Based on the preliminary composition elements and the recommended composition scheme, a voice guidance information is generated through a voice guidance engine. The voice guidance information includes operation instructions, principle explanations, and expected results. The voice guidance information is broadcast using emotional speech synthesis technology.

[0012] Furthermore, the aforementioned AI-based photography assistance method also includes: The system acquires the image content actually taken by the user, compares the image content with the recommended composition scheme, calculates the user's execution deviation, and collects user feedback data on the image content. Based on the execution deviation and the feedback data, the parameters of the user intent prediction model are optimized through a reinforcement learning algorithm, so that the user intent prediction model forms a sub-model that adapts to the user's personalized preferences.

[0013] Secondly, embodiments of this application provide an artificial intelligence-based photography assistance system, comprising: The first processing module is used to capture the viewfinder video stream in real time on the user device's camera viewfinder interface and simultaneously obtain multimodal data through the user device's device sensors. The second processing module is used to perform scene recognition and subject extraction based on the framing video stream using an image recognition model to obtain preliminary composition elements. The third processing module is used to generate at least one candidate mapping scheme based on the preliminary mapping elements and the multimodal dataset, using a user intent prediction model. The fourth processing module is used to score the aesthetic quality of multiple candidate composition schemes based on a pre-trained aesthetic evaluation model and select a recommended composition scheme. The fifth processing module is used to generate composition guide lines and output voice guidance information based on the recommended composition scheme in the photo framing interface to guide the user in taking photos.

[0014] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described artificial intelligence-based photography assistance method.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium, including a computer program stored thereon, which, when executed by a processor, implements the aforementioned artificial intelligence-based photography assistance method.

[0016] The beneficial effects of the embodiments in this application compared with the prior art are: This application discloses an AI-based photography assistance method that advances the assistance timing from "after shooting" to "during shooting," proactively providing guidance and offering higher decision support value. Furthermore, during the assistance process, by fusing multimodal data such as the viewfinder video stream and device sensors, and through deep learning aesthetic evaluation, the recommended solutions possess both scene relevance and professional quality. Simultaneously, by generating composition guidance lines in the photo framing interface and providing voice prompts, abstract photographic theory is transformed into intuitive and actionable guidance, enhancing the user experience. Additionally, a continuous learning mechanism based on bias analysis and reinforcement learning is introduced, enabling each model to adapt to and serve the user's personalized requirements, thereby increasing user engagement and the practicality of the technology. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a photography assistance method based on artificial intelligence, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of an artificial intelligence-based photography assistance system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0019] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0020] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0021] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0023] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0025] Please see Figure 1As shown, this invention is an AI-based photography assistance method that advances the assistance timing from "after shooting" to "during shooting," proactively providing guidance and offering higher decision support value. Furthermore, during the assistance process, through the fusion of multimodal data such as the viewfinder video stream and device sensor data, combined with deep learning aesthetic evaluation, the recommended solution possesses both scene relevance and professional quality. Specifically, it includes the following steps: S100: Capture the viewfinder video stream in real time on the user device's camera viewfinder interface, and simultaneously obtain multimodal data through the user device's device sensors. In some embodiments, step S100 above includes: The viewfinder video stream is captured in real time using the user's device camera and preset video display attributes; Based on the inertial measurement unit of the user device, the system continuously acquires raw motion data of the user device. By analyzing the raw motion data, the user's holding posture and the motion state of the shot are determined. Based on the ambient light sensor and color temperature sensor of the user device, the ambient lighting conditions are collected, and the lighting conditions are combined with the viewfinder video stream for analysis to determine the lighting direction and light quality of the shooting scene.

[0026] In this embodiment, multiple hardware sensors are activated in parallel on the user device's camera viewfinder, including a camera, an inertial measurement unit (IMU), an ambient light sensor, and a color temperature sensor. In some embodiments, the multi-source data obtained by the above sensors are timestamped and interpolated, and spatiotemporally synchronized to form a joint feature vector, which serves as the input basis for the user's subsequent models.

[0027] Specifically, the camera is used to capture real-time video streams for scene content recognition. The inertial measurement unit (IMU) continuously reads the device's raw motion data through its three-axis accelerometer, three-axis gyroscope, and three-axis magnetometer. Based on this raw motion data, a sensor fusion algorithm calculates the Euler angles of the device relative to the direction of gravity to determine the user's grip posture. Specifically, because the IMU is susceptible to various noise interferences during data acquisition, the raw data output from the three-axis accelerometer and three-axis gyroscope in the IMU needs to be preprocessed, including data removal and sliding window mean filtering. The preprocessed accelerometer and gyroscope data are then input into a sensor fusion algorithm based on extended Kalman filtering. Using the gravitational acceleration vector as the observation, this algorithm continuously estimates and outputs a quaternion representation of the device's coordinate system relative to the geographic coordinate system. Specifically, the quaternion represents the device's rotational attitude. This involves a combined rotation around the X / Y / Z axes, ultimately converting the quaternions into Euler angles to obtain the device's pitch, roll, and yaw angles. Then, based on the absolute value of the roll angle, supplemented by the range of the pitch angle, a preset attitude decision rule is used to determine whether the device is held in a landscape, portrait, or specific tilt orientation. For example, in one embodiment, when the roll angle ∈ [0°, 35°] ∪ [145°, 180°], it indicates that the user is holding the device vertically. For portrait orientation, when the roll angle is ∈ [55°, 125°], it indicates that the user is holding the device horizontally, which is a landscape orientation. When the roll angle is ∈ (35°, 55°) ∪ (125°, 145°) and the tilt angle indicates that the device screen is roughly facing upwards (not inverted), it is determined that the user may be intentionally tilting the creative composition, such as a diagonal composition. In addition, when the tilt angle is large, even if the roll angle is within the landscape orientation range, it may be judged as an unconventional posture to avoid misjudgment.

[0028] More specifically, the data patterns of the accelerometer and gyroscope are analyzed to distinguish whether the user is holding the device steadily, slowly translating, rapidly sweeping, or experiencing unexpected shaking during shooting. The identification of the shooting motion state can trigger different auxiliary strategies, such as providing fine composition lines when holding the device steadily and providing motion trajectory prediction when translating. Specifically, within a preset time window, time-domain and frequency-domain features are extracted from the preprocessed accelerometer and gyroscope data. The extracted time-frequency domain features are input into a time-series classification model, which is a hybrid model combining a convolutional neural network (CNN) and a long short-term memory (LSM) network. The CNN is used to extract local temporal patterns, and the LSM network is used to capture long-term dependencies. This time-series classification model can output the confidence level of the device in different motion states. Based on the confidence level, combined with state transition smoothing logic, the final motion state category of the device is determined. Specifically, the state transition smoothing logic involves pre-setting a state transition cost matrix, which defines the confidence threshold required to switch from one state to another. When the newly identified state is different from the previous state, the state switch is only performed if the confidence of the new state exceeds the sum of the confidence of the original state and the transition cost. Furthermore, based on the final motion state category, a corresponding photography assistance strategy is triggered. In one embodiment, when the state is identified as a stable holding state, fine composition guide lines are rendered on the viewfinder. When the state is identified as a slow panning state, motion trajectory prediction lines are overlaid on the viewfinder, and the position of the composition guide lines is dynamically adjusted. When the state is identified as a rapid sweeping state, a panoramic shooting assistance mode is enabled, and the panoramic synthesis progress and level maintenance guidance are displayed on the interface. When the state is identified as an unexpected shaking state, visual or tactile prompts are provided, and the user is advised to enable electronic image stabilization or use a tripod.

[0029] Furthermore, by using an ambient light sensor and a color temperature sensor, the illuminance and color temperature values ​​of the environment are collected to generate the lighting conditions of the shooting environment. The illuminance values ​​are then mapped to discrete lighting condition levels. Combined with the viewfinder video stream, the relationship between the brightness of the main area in the image and the lighting conditions is analyzed to jointly infer the lighting direction and quality of the shooting scene. Specifically, the raw illuminance and color temperature values ​​collected by the ambient light sensor and the color temperature sensor are calibrated and filtered. Based on the calibrated illuminance values ​​and a preset basic illuminance threshold range, the lighting conditions of the current environment are determined. In some embodiments, the basic illuminance threshold range is as follows: when the illuminance value ∈ [0, 10], it indicates a night scene; when the illuminance value is (10, 1000], it indicates indoor or low light conditions; when the illuminance value is (1000, 10000], it indicates shade or cloudy conditions; when the illuminance value > 10000, it indicates a low light condition. Lux indicates strong sunlight. Further, for the calibrated color temperature value, combined with lighting conditions, the dominant light source type of the current environment is determined. Dominant light source types include sunlight, moonlight, high-intensity artificial light, warm color temperature artificial light, and cool color temperature artificial light. Then, the lighting level and dominant light source type are combined with the viewfinder video stream in subsequent steps for analysis. For example, when the subject is identified as a task and the lighting conditions are strong sunlight, but the brightness of the subject's facial area in the image is insufficient, it can be jointly inferred that the current scene is backlit. The results obtained from the joint analysis can be used to assist in generating photo-taking guidance, such as generating specific voice guidance like "It is recommended to use a flash to fill in the light" or "Find a front lighting angle."

[0030] S200. Based on the framing video stream, an image recognition model is used to perform scene recognition and subject extraction to obtain preliminary composition elements; In some embodiments, step S200 above includes: Each frame of the viewfinder video stream is preprocessed to obtain a standardized image frame; The standardized image frames are input into a pre-trained image recognition model, and the image recognition model is used to perform scene classification and semantic segmentation to obtain scene classification results and semantic segmentation maps. Based on the scene classification results, dynamic subject filtering is performed, and the main visual subjects are identified and located by combining the semantic segmentation map. The scene classification results, semantic segmentation map, and main visual subjects are associated and encapsulated to form preliminary composition elements.

[0031] In this embodiment, to ensure the efficiency and consistency of subsequent model processing, each frame in the viewfinder video stream needs to be standardized preprocessed. Specifically, this includes scaling the image size to the fixed input size of the image recognition model, such as 224×224 or 320×320, normalizing the pixel values ​​to the range of [0,1], and adjusting the channel order, such as converting RGB to BGR. The processed standardized image frames are then fed into the pre-trained image recognition model for content recognition of the image frames.

[0032] In this embodiment, the image recognition model is a dual-branch network that integrates a temporal consistency module and a task interaction mechanism. Through this dual-branch network, scene classification and semantic segmentation tasks are executed in parallel. The scene classification branch integrates a channel attention module and receives a spatial attention map from the semantic segmentation branch as inter-task guidance information to focus on the scene-related global context. Specifically, in the scene classification branch, not only is the attention mechanism used in the channel dimension, but guidance information from the semantic segmentation branch is also introduced. Specifically, a spatial attention map is extracted from the intermediate layer of the semantic segmentation branch. This spatial attention map highlights key scene information that may be contained in the image, such as the horizon, building outlines, and large areas of vegetation. This spatial attention map is then weighted and superimposed on the feature map of the scene classification branch, making the scene classification branch pay more attention to semantically important areas rather than processing the entire image in a scattered manner. For example, when judging a "beach" scene, the model will more consciously combine the features of the boundary area between "sky," "sand," and "seawater," thereby improving the accuracy and robustness of the classification. In the semantic segmentation branch, a temporal consistency module is introduced at the decoder stage. This module uses optical flow information to align and fuse the segmentation features of the current frame and the previous frame to smooth the segmentation result and suppress transient jitter. Specifically, it calculates the dense optical flow field between the current frame and the previous frame. Based on this dense optical flow field, the high-level features of the previous frame in the decoder are reverse-distorted to align them spatially with the current frame. The aligned features of the previous frame and the current frame are then adaptively weighted and fused. The fusion weight is dynamically determined by the signal-to-noise ratio (SNR) of the current frame features. When the current frame has high quality and a high SNR, it is given a larger weight. When the quality of the current frame degrades due to motion blur or noise interference, the relatively stable historical features aligned from the previous frame are relied upon more. This effectively suppresses boundary flicker and category jump phenomena common in real-time video segmentation, outputting a temporally smooth semantic segmentation map.

[0033] More specifically, by pre-setting a subject priority mapping table associated with scene categories, when the image recognition model outputs the corresponding scene classification result, it queries the subject priority mapping table to obtain the priority weight of each category of subjects under the current scene classification result, and generates a weight vector. For example, when the scene is classified as "natural scenery", "mountains" and "water bodies" have higher priority than the occasional "people"; when the scene is classified as "urban street photography", "people", "unique means of transportation", and "landmark buildings" have the highest priority; when the scene is classified as "indoor portrait", "people" have absolute priority.

[0034] Furthermore, in the semantic segmentation graph, all connected components belonging to the preset subject category are filtered out, and the physical pixel area of ​​each connected component is calculated. For each connected component, according to its category, its corresponding category priority weight is obtained from the weight vector, thereby calculating its comprehensive saliency score. Specifically, the comprehensive saliency score is a function of the physical pixel area, the category priority weight, and the spatial location weight. Specifically, S_i=(A_i / A_max)×W_{C_i}×L_i, where S_i represents the comprehensive saliency score of connected component i, A_i represents the physical pixel area of ​​connected component i, A_max represents the maximum area among all connected components in the current image frame, W_{C_i} represents the category priority weight of the category corresponding to connected component i, and L_i represents the spatial location weight, which is calculated based on the distance between the centroid of the connected component and the center of the image. In one embodiment, a two-dimensional Gaussian function model is used to calculate L_i. This function takes the center of the image as the mean and sets a standard deviation σ. When the centroid of the connected component (x_i, ...) is at a distance of 0.05, ... The closer y_i is to the center of the image (x_c, y_c), the closer the L_i value is to 1; the farther the distance between them, the more exponentially the L_i value decays. Therefore, the formula for spatial position weight can be approximately expressed as: L_i=exp(-( (x_i-x_c)^2+(y_i-y_c)^2 ) / (2×σ^2)); finally, the connected component with the highest comprehensive significance score S_i is determined as the main visual subject of the current frame.

[0035] More specifically, the position coordinates of the main visual subject in the image are combined with the raw motion data obtained by the inertial measurement unit (IMU) in step S100 to calculate the relative spatial orientation of the main visual subject in the real world. Specifically, the image coordinates (u, v) of the main visual subject are combined with the attitude angles of the user device, namely pitch angle and yaw angle, and the spatial orientation of the subject relative to the optical axis of the device is estimated through a projection geometry model. At the same time, the scene classification result is associated with the lighting conditions obtained by the ambient light sensor and color temperature sensor in step S100 to jointly form a description of the environmental conditions. For example, in one embodiment, the "night scene" scene classification result is associated with extremely low ambient illuminance value, or the inference of "backlight scene" is associated with high contrast lighting conditions. This ultimately forms preliminary composition elements to support the subsequent generation of shooting suggestions.

[0036] S300. Based on the preliminary mapping elements and combined with the multimodal dataset, at least one candidate mapping scheme is generated using a user intent prediction model. In some embodiments, step S300 above includes: The initial mapping elements are structurally associated with the multimodal data to construct a multimodal intent understanding map; The multimodal intent understanding graph is input into the user intent prediction model. The user intent prediction result is obtained through the user intent prediction model and retrieved and reasoned in the graph composition scheme knowledge graph to generate multiple candidate graph composition schemes. For each candidate graphing scheme, parameter instantiation is performed in conjunction with the multimodal intent understanding graph to generate parameterized information.

[0037] In this embodiment, the multimodal intent understanding map includes visual subject nodes, scene element nodes, and device status nodes. The visual subject node is the main visual subject determined in step S200, and its attributes specifically include its category, bounding box, contour mask, and centroid coordinates. The scene element node specifically refers to scene elements selected from the semantic segmentation map, such as "sky area," "ground area," "water area," and "building area," and its attributes include its average color, texture, and proportion in the image. The device status node specifically refers to the user's device holding posture and the shooting motion state, such as "horizontal device holding," "vertical device holding," and "stable state." The attributes of "translation state" and "transition state" are specifically the raw motion data continuously obtained by the IMU inertial measurement unit in step S100. Further, the visual subject node and scene element node are connected to form a spatial relationship edge. The type and weight of the spatial relationship change represent the spatial relationship between the nodes, such as "subject-located in-foreground", "building-enclosed in-sky", etc. The relationship strength of the above spatial relationship is quantified by calculating the overlap of the bounding box or the centroid distance. At the same time, from the scene element node and the device status node, a semantic relationship edge is formed pointing to the virtual intention node. Specifically, a photography knowledge base is constructed, which contains a series of production rules of IF <condition> THEN <intention>. For example, in one embodiment, there is a rule: IF (existence of person node AND device posture = vertical holding) THEN (intention = highlighting the subject). The rule condition is that if there is a "person" node in the image frame and the device status node is displayed as "vertical holding", then a semantic logic edge is established between the "person" node and the "intention" node, and between the "device status" node and the "intention" node, and a high initial weight is assigned according to the rule confidence.

[0038] More specifically, the constructed multimodal intent understanding graph is input into the user intent prediction model. In the user intent prediction model, a graph attention network is used to update the node features of the multimodal intent understanding graph, so that the feature vector of each node contains the contextual information of its neighboring nodes. At the same time, through a graph-level readout mechanism, the information of the entire graph is aggregated into a global intent embedding vector. Then, the global intent embedding vector is input into a multilayer perceptron, which outputs a probability distribution in a predefined intent space, including but not limited to highlighting the subject, showing the environment, creating a sense of depth, and expressing symmetry. The user intent prediction result is determined based on the probability distribution result.

[0039] Furthermore, based on the user intent prediction results, retrieval and reasoning are performed in the composition scheme knowledge graph to generate a set of candidate composition schemes with priority ranking. The composition scheme knowledge graph stores various composition rules in the form of nodes, such as the "rule of thirds," "symmetrical composition," "leading line composition," and "frame composition." Each composition rule node is associated with applicable scene types, suitable shooting intentions, and typical aesthetic effects through edges. Scene types include "portrait," "landscape," and "architecture," shooting intentions include highlighting the subject, showcasing the environment, creating a sense of depth, and expressing symmetry, and aesthetic effects include "balance," "dynamism," and "stability." Further, the scene classification results obtained in step S200 and the aforementioned user intent prediction results are used as joint query conditions for vector similarity retrieval in the composition scheme knowledge graph. For the retrieved composition rule nodes, their matching degree with the query conditions and their prior weights in the composition scheme knowledge graph are considered to calculate the similarity of each composition rule node. The final score of the composition scheme is used to prioritize the composition schemes. For example, in one embodiment, it is necessary to search the composition scheme knowledge graph for composition rules that satisfy "highlighting the subject" and "indoor portrait". During the search, composition rules that satisfy the above scene classification results and user intent prediction results will be retrieved first. For example, "rule of thirds" is applicable to "portrait" and serves "highlighting the subject", thus obtaining a candidate composition scheme. In addition, multi-hop reasoning is performed along the edges in the composition scheme knowledge graph. For example, the search finds that "frame composition" also serves "highlighting the subject", but it is not directly marked as applicable to "indoor". However, there may be elements that can be used as frames in "indoor" scenes, such as doors and windows. Through the above multi-hop reasoning, more potential composition rules that can be applied to the current shooting scene and intent are mined. Finally, by combining the matching degree and prior knowledge, a list of candidate composition schemes with ordered positions is output. The prior knowledge specifically refers to the universality and effect of certain composition rules in different scenes.

[0040] More specifically, for each candidate composition scheme, the attributes of each node in the image are combined with the aforementioned multimodal intent understanding to perform parameterized instantiation, thereby generating parameterized information. This transforms the abstract composition rules into concrete guide lines on the user's device that perfectly match the camera viewfinder. For example, for the "rule of thirds" composition scheme, the centroid of the visual subject node is calculated based on its contour mask, and the key points of the composition guide line, such as the four intersection points, are spatially associated with the centroid of the visual subject node. The precise position of the guide line in the image is dynamically adjusted to obtain the corresponding spatial coordinates of the guide line, which are then used as parameterized information for the subsequent generation of guide lines. For the "guide line" composition scheme, scene elements with linear characteristics, such as roads, rivers, and railings, are identified in the semantic segmentation image. The directions and endpoints of these scene elements with linear characteristics are extracted using Hough transform or line segment detection algorithms to generate dominant direction vectors, which are then used as parameterized information for the subsequent generation of guide lines.

[0041] S400. Based on a pre-trained aesthetic evaluation model, the aesthetic quality of multiple candidate composition schemes is scored, and a recommended composition scheme is selected. In some embodiments, step S400 above includes: Based on the composition rules of each candidate composition scheme, image simulation is performed on the original viewfinder frame to generate a composition reference image of the corresponding candidate composition scheme. The composition reference image is input into a pre-trained aesthetic evaluation model, and the aesthetic evaluation model outputs an overall aesthetic score and a multi-dimensional aesthetic feature vector. The multidimensional aesthetic feature vectors are dynamically matched with the user intent prediction results, and the intent conformity weighted score of each candidate solution is calculated. By combining the overall aesthetic score and the intention conformity weighted score, a final comprehensive score is obtained for each candidate composition scheme, and a recommended composition scheme is determined based on the final comprehensive score.

[0042] In this embodiment, for each candidate composition scheme, virtual rendering is performed on the original view frame according to its composition rules to generate a composition reference image for each candidate composition scheme. Specifically, according to the composition rule type of the candidate composition scheme, the corresponding image transformation model and rendering pipeline are selected to render the candidate composition scheme into the corresponding composition reference image. For example, for composition schemes involving image cropping and translation, an affine transformation model is selected for rendering; for composition schemes involving perspective simulation, a perspective projection transformation model is selected; and for composition schemes involving depth simulation, a depth rendering pipeline is enabled, which includes foreground-background separation and background blurring processing of the scene depth map.

[0043] More specifically, the composition reference images of each candidate composition scheme are input into a pre-trained aesthetic evaluation model. This model can output an overall aesthetic score and a multi-dimensional aesthetic feature vector in parallel. Specifically, the aesthetic evaluation model includes a backbone network, an overall aesthetic regression branch, and an aesthetic attribute branch. The backbone network is a feature extractor used to extract composition features from the composition reference images as shared features for the overall aesthetic regression branch and the aesthetic attribute branch. The overall aesthetic regression branch and the aesthetic attribute branch run in parallel. The overall aesthetic regression branch is a fully connected layer used to output the overall aesthetic score, representing the overall aesthetic level of the composition reference image. The aesthetic attribute branch is further divided into multiple sub-networks, each responsible for an aesthetic dimension. The model performs aesthetic evaluations and quantifies them to obtain corresponding aesthetic dimension scores, generating multi-dimensional aesthetic feature vectors. These aesthetic dimensions include compositional harmony, visual balance, subject prominence, color harmony, and appropriate depth of field. During the training phase, fine-grained labels for each aesthetic dimension, manually annotated from a professional photography review dataset, are used as supervision signals. Within the model, an independent attention map generation module is configured for each aesthetic dimension. This module can be used to visualize the key image regions that are of interest when evaluating the corresponding aesthetic dimension, highlighting which regions contribute the most to the evaluation. This makes the evaluation process of the aesthetic evaluation model for the compositional reference image transparent and traceable.

[0044] Furthermore, based on the multi-dimensional aesthetic feature vector and combined with the user intent prediction results, a dynamic weight allocation strategy is used to calculate the intent compliance weighted score for each candidate solution. Specifically, an intent-aesthetic dimension weight mapping table is predefined, which defines the importance weight of each aesthetic dimension under different shooting intents. For example, in one embodiment, when the dominant intent is "highlighting the subject", the subject prominence has the highest weight, followed by compositional harmony, while color harmony and depth of field appropriateness also have certain weights; when the dominant intent is "showing the environment", the weights of visual balance and color harmony increase, while the weight of subject prominence decreases; when the dominant intent is "creating a sense of depth", depth of field appropriateness and compositional harmony have the highest weights. Thus, based on the user intent prediction results, the above aesthetic dimension weight mapping table is queried to obtain a set of dynamic weights, and the multi-dimensional aesthetic feature vector is weighted and summed to obtain the intent compliance weighted score.

[0045] More specifically, the overall aesthetic score and the weighted score of intent conformity are combined according to a preset ratio to obtain the final comprehensive score of each candidate composition scheme. Finally, the candidate composition scheme with the highest score is selected as the recommended composition scheme.

[0046] S500: On the camera viewfinder, based on the recommended composition scheme, a composition guide line is generated and voice guidance information is output to guide the user in taking photos.

[0047] In this application, by generating composition guide lines in the photo framing interface and providing voice prompts, abstract photography theory is transformed into intuitive and operable guidance, thereby improving the user experience.

[0048] In some embodiments, step S500 above includes: Based on the parametric information corresponding to the recommended composition scheme, dynamic composition guide lines are generated and rendered in the photo framing interface. The composition guide lines include static frame lines and dynamic semantic lines, wherein the static frame lines are used to represent the basic structure of the composition, and the dynamic semantic lines are used to highlight specific visual elements in the scene. Based on the preliminary composition elements and the recommended composition scheme, a voice guidance information is generated through a voice guidance engine. The voice guidance information includes operation instructions, principle explanations, and expected results. The voice guidance information is broadcast using emotional speech synthesis technology.

[0049] In this embodiment, a screen coordinate system is constructed in the camera viewfinder, with its origin at the upper left corner or center of the screen, set according to actual needs. Then, based on the parameterized information of the recommended composition scheme, the position and shape of the static composition frame line are calculated in the screen coordinate system. For dynamic semantic guide lines, the contours or dominant edges of the two-dimensional image elements identified by the semantic segmentation map are directly mapped to guide line paths in the screen coordinate system.

[0050] In some embodiments, a motion compensation mechanism is established within the screen space. By fusing motion data from the device's IMU (Inertial Measurement Unit), the position of the static frame lines on the screen can resist translation and rotation caused by unexpected user jitter, thereby maintaining visual stability.

[0051] More specifically, based on the initial composition elements and recommended composition scheme, a voice guidance engine based on speech synthesis and natural language generation technology generates structured voice guidance information. Specifically, the voice guidance engine has an embedded voice script template library, which can fill and combine variables in the template in real time according to the recommended composition scheme, initial composition elements, and user intent prediction results. For example, in one embodiment, there is a voice script template "[Operation Instruction] Please move [Subject Name] to [Target Position]. [Principle Explanation] This conforms to [Composition Rule Name]. [Expected Effect] It can effectively achieve [Aesthetic Effect]." When generating voice guidance information, variables are dynamically filled according to the scene content in the initial composition elements and the recommended composition scheme, such as Subject Name = "Tower", Target Position = "Right Thirds Line", Composition Rule Name = "Right Thirds", Aesthetic Effect = "Highlight Subject", thereby generating natural and fluent guidance statements.

[0052] Furthermore, emotional speech synthesis technology is used for broadcasting, and short prompt sound effects are inserted before key operation commands. For example, TTS technology that supports multiple tones and rhythms is used. When broadcasting the principles and effects, a calm and explanatory tone is used; and before issuing key operation commands, a prompt sound effect is inserted, and the tone may be slightly raised to grab the user's attention.

[0053] In some embodiments, while rendering the composition guide lines, the aesthetic dimension scores and explanations, derived from the aesthetic evaluation model and corresponding to the current guide lines, are simultaneously displayed on the side of the camera viewfinder. These scores may be displayed as five stars or progress bars, representing aesthetic dimensions such as compositional harmony and subject prominence. Simultaneously, brief text explanations are provided for the scores, such as "+Good subject position" or "-The horizon line is slightly slanted." This allows users to immediately understand the aesthetic quality of the current composition during shooting, while also grasping the corresponding aesthetic principles. This transforms a simple photo-taking assistance exercise into a learning process of photographic knowledge, thereby improving shooting quality.

[0054] In some embodiments, the visual state and voice guidance content of the guide line can be dynamically updated based on the user's real-time operations and screen changes to achieve interactive guidance. For example, in the initial / unsatisfactory state, the guide line is displayed as a semi-transparent red or yellow, indicating that adjustment is needed. In the near-target state, that is, when the user's operation brings the screen composition closer to the recommended solution, the guide line becomes a jumping animation, that is, it changes its visual attributes, such as brightness or transparency, in a rhythmic and periodic manner. In the satisfactory state, that is, when the composition fully meets the requirements, the guide line becomes a bright and stable green. Furthermore, the voice guidance will also be adjusted accordingly in conjunction with the changes in visual state. For example, when entering the "near-target state," it will announce "Very good, close!"; when entering the "satisfactory state," it will announce "Perfect composition! You can prepare to shoot." Through immediate and positive feedback, the user's sense of accomplishment and interactive enjoyment are enhanced.

[0055] In some embodiments, the above-described AI-based photography assistance method further includes: The system acquires the image content actually taken by the user, compares the image content with the recommended composition scheme, calculates the user's execution deviation, and collects user feedback data on the image content. Based on the execution deviation and the feedback data, the parameters of the user intent prediction model and / or the aesthetic evaluation model are optimized through reinforcement learning algorithms, so that the user intent prediction model and / or the aesthetic evaluation model form a sub-model that adapts to the user's personalized preferences.

[0056] This application also introduces a continuous learning mechanism based on bias analysis and reinforcement learning, enabling each model to adapt to and serve the user's personalized requirements, thereby enhancing user stickiness and the practicality of the technology.

[0057] In this embodiment, the overlap degree between the position of the main subject in the image content actually captured by the user and the connected components of the main visual subject in the recommended composition scheme is calculated. At the same time, the tilt angle of the horizontal line in the actual captured image and the angle difference between the horizontal line in the recommended composition scheme are calculated. The overlap degree and the angle difference are weighted and fused to obtain the execution deviation degree.

[0058] Furthermore, user feedback data on the actual captured image content is collected, including explicit and implicit feedback. For explicit feedback, the user's numerical rating of the image content, such as a five-star rating, can be directly mapped to a normalized value. For implicit feedback, the user's textual feedback on the image content, such as a text comment like "I really like this photo," is processed through a sentiment analysis model to output a sentiment polarity score as feedback data.

[0059] Furthermore, the execution bias and feedback data are concatenated to form the environment state for reinforcement learning, and a reward function is constructed based on this to guide the learning direction. Specifically, R(s,a)=w1×S_explicit+w2×S_sentiment-w3×D, where s represents the environment state, a represents the adjustment of the parameters of the user intent prediction model and / or aesthetic evaluation model, S_explicit represents explicit feedback, S_sentiment represents implicit feedback, D represents the execution bias, w1, w2, and w3 represent weight coefficients, and w1+w2+w3=1.

[0060] Please see Figure 2 As shown, the present invention also provides an artificial intelligence-based photography assistance system, the system comprising: First processing module 201: used to capture the viewfinder video stream in real time on the user device's viewfinder interface, and simultaneously obtain multimodal data through the user device's device sensors; The second processing module 202 is used to perform scene recognition and subject extraction based on the framing video stream using an image recognition model to obtain preliminary composition elements; The third processing module 203 is used to generate at least one candidate mapping scheme based on the preliminary mapping elements and the multimodal dataset, using a user intent prediction model. The fourth processing module 204 is used to score the aesthetic quality of multiple candidate composition schemes based on a pre-trained aesthetic evaluation model and select a recommended composition scheme. The fifth processing module 205 is used to generate composition guide lines and output voice guidance information on the photo framing interface based on the recommended composition scheme to guide the user in taking photos.

[0061] It is understandable that, such as Figure 1 The content of the AI-based photography assistance method embodiments shown herein is applicable to the AI-based photography assistance system embodiments. The specific functions implemented by the AI-based photography assistance system embodiments are the same as those shown below. Figure 1 The illustrated embodiment of the AI-based photography assistance method is the same, and the beneficial effects achieved are the same as those shown. Figure 1 The beneficial effects achieved by the illustrated AI-based photography assistance method embodiment are the same.

[0062] It should be noted that the information interaction and execution process between the above systems are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0063] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0064] Please see Figure 3 As shown, this embodiment of the invention also provides a computer device 3, including: a memory 302 and a processor 301, and a computer program 303 stored on the memory 302. When the computer program 303 is executed on the processor 301, it implements the artificial intelligence-based photography assistance method as described in any of the above methods.

[0065] The computer device 3 may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will understand that... Figure 3 The computer device 3 is merely an example and does not constitute a limitation on the computer device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0066] The processor 301 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0067] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may be an external storage device of the computer device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 3. Furthermore, the memory 302 may include both internal and external storage units of the computer device 3. The memory 302 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0068] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the AI-based photography assistance method as described in any of the above methods.

[0069] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / computer device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0070] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A photography-assisted method based on artificial intelligence, characterized in that, include: On the user device's camera viewfinder, capture the viewfinder video stream in real time and simultaneously obtain multimodal data through the user device's sensors; Based on the framing video stream, an image recognition model is used to perform scene recognition and subject extraction to obtain preliminary composition elements; Based on the preliminary mapping elements and combined with the multimodal dataset, at least one candidate mapping scheme is generated using a user intent prediction model. Based on a pre-trained aesthetic evaluation model, the aesthetic quality of multiple candidate composition schemes is scored, and a recommended composition scheme is selected. In the photo framing interface, based on the recommended composition scheme, composition guide lines are generated and voice guidance information is output to guide the user in taking photos.

2. The method as described in claim 1, characterized in that, The process of capturing a viewfinder video stream in real time on the user device's camera viewfinder interface and simultaneously obtaining multimodal data through the user device's sensors includes: The viewfinder video stream is captured in real time using the user's device camera and preset video display attributes; Based on the inertial measurement unit of the user device, the system continuously acquires raw motion data of the user device. By analyzing the raw motion data, the user's holding posture and the motion state of the shot are determined. Based on the ambient light sensor and color temperature sensor of the user device, the ambient lighting conditions are collected, and the lighting conditions are combined with the viewfinder video stream for analysis to determine the lighting direction and light quality of the shooting scene.

3. The method as described in claim 1, characterized in that, Based on the framing video stream, an image recognition model is used to perform scene recognition and subject extraction to obtain preliminary composition elements, including: Each frame of the viewfinder video stream is preprocessed to obtain a standardized image frame; The standardized image frames are input into a pre-trained image recognition model, and the image recognition model is used to perform scene classification and semantic segmentation to obtain scene classification results and semantic segmentation maps. Based on the scene classification results, dynamic subject filtering is performed, and the main visual subjects are identified and located by combining the semantic segmentation map. The scene classification results, semantic segmentation map, and main visual subjects are associated and encapsulated to form preliminary composition elements.

4. The method as described in claim 1, characterized in that, The step of generating at least one candidate mapping scheme based on the preliminary mapping elements and the multimodal data using a user intent prediction model includes: The initial mapping elements are structurally associated with the multimodal data to construct a multimodal intent understanding map; The multimodal intent understanding graph is input into the user intent prediction model. The user intent prediction result is obtained through the user intent prediction model and retrieved and reasoned in the graph composition scheme knowledge graph to generate multiple candidate graph composition schemes. For each candidate graphing scheme, parameter instantiation is performed in conjunction with the multimodal intent understanding graph to generate parameterized information.

5. The method as described in claim 1, characterized in that, The pre-trained aesthetic evaluation model scores the aesthetic quality of multiple candidate composition schemes and selects a recommended composition scheme, including: Based on the composition rules of each candidate composition scheme, image simulation is performed on the original viewfinder frame to generate a composition reference image of the corresponding candidate composition scheme. The composition reference image is input into a pre-trained aesthetic evaluation model, and the aesthetic evaluation model outputs an overall aesthetic score and a multi-dimensional aesthetic feature vector. The multidimensional aesthetic feature vectors are dynamically matched with the user intent prediction results, and the intent conformity weighted score of each candidate solution is calculated. By combining the overall aesthetic score and the intention conformity weighted score, a final comprehensive score is obtained for each candidate composition scheme, and a recommended composition scheme is determined based on the final comprehensive score.

6. The method as described in claim 1, characterized in that, In the photo-taking interface, based on the recommended composition scheme, composition guide lines are generated and voice guidance information is output to guide the user in taking photos, including: Based on the parametric information corresponding to the recommended composition scheme, dynamic composition guide lines are generated and rendered in the photo framing interface. The composition guide lines include static frame lines and dynamic semantic lines, wherein the static frame lines are used to represent the basic structure of the composition, and the dynamic semantic lines are used to highlight specific visual elements in the scene. Based on the preliminary composition elements and the recommended composition scheme, a voice guidance information is generated through a voice guidance engine. The voice guidance information includes operation instructions, principle explanations, and expected results. The voice guidance information is broadcast using emotional speech synthesis technology.

7. The method as described in claim 1, characterized in that, Also includes: The system acquires the image content actually taken by the user, compares the image content with the recommended composition scheme, calculates the user's execution deviation, and collects user feedback data on the image content. Based on the execution deviation and the feedback data, the parameters of the user intent prediction model are optimized through a reinforcement learning algorithm, so that the user intent prediction model forms a sub-model that adapts to the user's personalized preferences.

8. A photography assistance system based on artificial intelligence, characterized in that, include: The first processing module is used to capture the viewfinder video stream in real time on the user device's camera viewfinder interface and simultaneously obtain multimodal data through the user device's device sensors. The second processing module is used to perform scene recognition and subject extraction based on the framing video stream using an image recognition model to obtain preliminary composition elements. The third processing module is used to generate at least one candidate mapping scheme based on the preliminary mapping elements and the multimodal dataset, using a user intent prediction model. The fourth processing module is used to score the aesthetic quality of multiple candidate composition schemes based on a pre-trained aesthetic evaluation model and select a recommended composition scheme. The fifth processing module is used to generate composition guide lines and output voice guidance information based on the recommended composition scheme in the photo framing interface to guide the user in taking photos.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images