Scenic spot tourist non-sensing snapshot and intelligent optimization system based on multi-modal perception

Abstract

The application discloses a scenic spot tourist non-sensing snapshot and intelligent optimization system based on multi-modal perception and relates to the technical field of intelligent control and optimization. Multi-modal perception data are acquired through a wide-angle and a holder camera. Firstly, the sensitivity of camera parameter adjustment to composition indexes is evaluated, and a reference visual angle satisfying a landmark integrity hard constraint is generated. Then, taking the reference parameter as an initial value, small-step perturbation iterative optimization is carried out around soft indexes such as a figure blocking rate, a frame proportion and an aesthetic score. Finally, the stability of the optimization result is checked, and a snapshot is triggered, so as to form an online closed-loop learning mechanism. The application realizes automatic and high-quality figure and landmark collaborative composition in a scenic spot scene, and effectively improves the composition quality and visual aesthetic feeling of a tourist snapshot.

Description

Technical Field

[0001] This invention relates to the field of intelligent control and optimization technology, and more specifically, to a multimodal perception-based system for seamless capture and intelligent optimization of tourists in scenic areas. Background Technology

[0002] Currently, photography services offered to tourists in scenic areas largely rely on manual shooting or self-service operation, resulting in inefficiency, inconsistent composition quality, and difficulty in simultaneously ensuring the integrity of landmarks and the aesthetic appeal of portraits. While some automated shooting systems can detect and capture tourists, they typically lack deep scene perception and intelligent composition capabilities, often failing to achieve professional-level results. Furthermore, existing systems generally lack collaborative optimization of the relationship between landmarks and tourists, failing to dynamically adjust shooting parameters in complex scenes to generate high-quality photos, and struggling to establish a closed-loop learning and continuous optimization mechanism based on practical shooting experience. Therefore, there is an urgent need for a scenic area tourist capture and intelligent optimization system that can achieve automation, high quality, and learning capabilities.

[0003] To address the above problems, this invention proposes a solution. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a multimodal perception-based system for seamless capture and intelligent selection of tourists in scenic areas, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] The system for seamless capture and intelligent selection of tourists in scenic areas based on multimodal perception includes: a motion sensitivity assessment module, a baseline viewpoint generation module, a residual composition optimization module, and a capture closed-loop fusion module, with signal connections between each module;

[0007] Action sensitivity assessment module: Extracts person instance masks, landmark key part masks, and structural key point sets from panoramic and close-up images; Combines camera intrinsic and extrinsic parameters, performs single-step micro-exploration of horizontal swing angle, pitch angle, and focal length respectively, calculates the difference in composition score, landmark integrity, and person occlusion rate, and obtains three-axis sensitivity; Based on this, sets the increment upper limit and step size to form a priority-arranged action candidate sequence;

[0008] The baseline view generation module uses the landmark integrity of not less than a threshold as a hard constraint. It converts the pixel deviation of the landmark outer frame into angle and field of view adjustment amount in the order of horizontal swing angle, pitch angle and focal length, and gradually corrects it until the boundary is complete and falls into the safe area of ​​the screen. If it is still insufficient, it iterates without exceeding the incremental upper limit to obtain the baseline camera parameter set that meets the hard constraint.

[0009] Residual composition optimization module: Using the baseline camera parameter set as the initial value, it refines the image by small perturbation steps with sensitivity decreasing from high to low, focusing on soft indicators such as the occlusion rate of people, the proportion of people in the frame, the visual center of gravity and the balance of white space. Each round only changes a single degree of freedom and verifies in real time that the integrity of landmarks does not decrease. When the gain is lower than the threshold or the iteration limit is reached, it stops and outputs the optimized camera parameter set and residual record.

[0010] The capture closed-loop fusion module performs consistency and stability checks on the benchmark solution and the optimized solution, and determines whether to capture based on the image shake amplitude, target tracking stability and focus sharpness threshold. After capturing, the panoramic channel is used to check the landmark integrity and the person occlusion rate and generate quality labels. The action sequence and step size selection are written into the experience cache, and the incremental upper limit, step size and priority of subsequent sensitivity estimation are updated to form an online closed loop.

[0011] In a preferred embodiment, the motion sensitivity assessment module includes the following steps:

[0012] Read the camera's intrinsic and extrinsic parameters and use the mask of human instance, the mask of key landmark parts, and the set of key structural points to calculate the elements of the image;

[0013] By applying positive and negative single-step micro-explorations to the three degrees of freedom of horizontal swing angle, pitch angle, and focal length, the changes in composition score, landmark integrity, and human occlusion rate were obtained.

[0014] Normalize the three-degree-of-freedom changes to form a sensitivity vector, generate and output the action candidate sequences containing degrees of freedom, step size and upper limit of increment in descending order of sensitivity.

[0015] In a preferred embodiment, the reference viewpoint generation module includes the following steps:

[0016] The horizontal and vertical pixel deviations of the landmark outer frame relative to the center of the image are converted into corresponding angular and field-of-view changes.

[0017] According to the priority of the action candidate sequence, horizontal sway angle correction, pitch angle correction and focal length correction are applied in sequence. After each correction, the landmark integrity is calculated and compared with the threshold.

[0018] If the threshold is not reached, the iteration continues without exceeding the increment limit and step size. When the threshold is reached and the landmark boundary is in the safe zone, the parameters are frozen and the reference camera parameter set is output.

[0019] In a preferred embodiment, the residual mapping optimization module includes the following steps:

[0020] Soft indicators such as the occlusion rate of people, the proportion of people in the frame, the visual center of gravity deviation, and the white space are calculated using the reference camera parameter set as the initial value.

[0021] Candidate solutions are obtained by applying small-step perturbations to a single degree of freedom, sorted by sensitivity.

[0022] For candidate solutions, the landmark integrity is checked to ensure it is not lower than the threshold. If the threshold is not met, the perturbation is rolled back and the step size is reduced before retrying.

[0023] Record the residual and soft index gain for each round. When the continuous gain is lower than the convergence threshold or reaches the iteration limit, output the optimized camera parameter set and residual record.

[0024] The occlusion rate of a person is calculated as the ratio of the intersection area of ​​the person instance mask and the landmark key part mask to the area of ​​the landmark key part mask.

[0025] In the residual optimization process, priority is given to perturbing the degree of freedom direction that makes the occlusion rate of the character monotonically decrease.

[0026] When occlusion rate oscillations occur, reduce the step size and maintain landmark integrity at or above the threshold.

[0027] The proportion of the figure in the frame is calculated as the ratio of the figure's outer frame area to the total frame area, and then compared with the expected proportion.

[0028] Based on the camera's current field of view, the proportional deviation is converted into a focal length change and proportional correction is performed.

[0029] After each proportional correction, the integrity of the landmark is evaluated simultaneously. If it is not satisfied, the correction is rolled back and the step size is reduced. At the same time, if necessary, a small horizontal swing angle or pitch angle correction is used to suppress the visual center of gravity shift.

[0030] Calculate the visual center of gravity for the main character area and the landmark area respectively, and synthesize the visual center of gravity of the image according to the weights. Calculate the deviation of the center of gravity from the center of the image and the white space on the four sides of the image.

[0031] Simultaneously, small-step perturbations are performed in the perturbation direction to reduce center of gravity deviation and improve the balance of white space.

[0032] When approaching the landmark integrity threshold boundary, reduce the step size until the safety margin is restored and record the corresponding residual.

[0033] In a preferred embodiment, the capture closed-loop fusion module includes the following steps:

[0034] Soft indicators such as the occlusion rate of people, the proportion of people in the frame, the visual center of gravity deviation, and the white space are calculated using the reference camera parameter set as the initial value.

[0035] Candidate solutions are obtained by applying small-step perturbations to a single degree of freedom, sorted by sensitivity.

[0036] For candidate solutions, the landmark integrity is checked to ensure it is not lower than the threshold. If the threshold is not met, the perturbation is rolled back and the step size is reduced before retrying.

[0037] Record the residual and soft index gain for each round. When the continuous gain is lower than the convergence threshold or reaches the iteration limit, output the optimized camera parameter set and residual record.

[0038] The panoramic view is used to calculate the landmark integrity and human occlusion rate of the entire area and generate quality labels.

[0039] Write the consistency check results, stability check results, actual action sequence and step size, and soft and hard index responses into the experience cache;

[0040] Based on experience caching, sensitivity statistics are updated online, and the increment limit, step size, and priority of action candidate sequences are adjusted accordingly for subsequent loop use.

[0041] When the optimized camera parameter set and the baseline camera parameter set exceed the limits in the consistency check and the stability is insufficient, the baseline camera parameter set or the most recently stable intermediate parameter set should be selected to perform the snapshot.

[0042] The gimbal and zoom lens are driven to perform minute angle and focal length movements according to the selected parameter group;

[0043] Before and after the action, assess the integrity of the landmark, the occlusion rate of the person, the proportion of the person in the frame, and the white space of the visual center of gravity, and record the execution log.

[0044] The technical effects and advantages of this invention are as follows: A multimodal perception-based system for seamless capture and intelligent selection of tourists in scenic areas.

[0045] Through multimodal perception and intelligent composition optimization, the system automatically completes collaborative composition and shooting of tourists and landmarks without human intervention, generating scenic area photos with professional composition quality, greatly enhancing the tourist experience and photo sharing value. Using landmark integrity as a hard constraint, it iterative optimization is performed using multiple soft indicators such as human composition, occlusion rate, and aesthetic score to ensure that each photo not only fully presents the scenic landmarks but also possesses good visual balance and compositional beauty. Through real-time perception and feedback mechanisms, camera parameters are dynamically adjusted to adapt to tourist movement and scene changes. Simultaneously, the action sequence and step length selection during the capture process are recorded and used to update subsequent strategies, forming an online closed-loop optimization that continuously improves system response speed and composition accuracy. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the structure of the scenic area tourist non-intrusive capture and intelligent selection system based on multimodal perception, according to the present invention.

[0047] Figure 2 This is a schematic diagram of the process of the non-intrusive capture and intelligent selection system for tourists in scenic areas based on multimodal perception, according to the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Example

[0050] Please see Figures 1-2 As shown, this invention discloses a non-intrusive capture and intelligent optimization system for tourists in scenic areas based on multimodal perception, including: a motion sensitivity assessment module, a reference view generation module, a residual composition optimization module, and a capture closed-loop fusion module, with signal connections between each module;

[0051] Action sensitivity assessment module: Extracts person instance masks, landmark key part masks, and structural key point sets from panoramic and close-up images; Combines camera intrinsic and extrinsic parameters, performs single-step micro-exploration of horizontal swing angle, pitch angle, and focal length respectively, calculates the difference in composition score, landmark integrity, and person occlusion rate, and obtains three-axis sensitivity; Based on this, sets the increment upper limit and step size to form a priority-arranged action candidate sequence;

[0052] The baseline view generation module uses the landmark integrity of not less than a threshold as a hard constraint. It converts the pixel deviation of the landmark outer frame into angle and field of view adjustment amount in the order of horizontal swing angle, pitch angle and focal length, and gradually corrects it until the boundary is complete and falls into the safe area of ​​the screen. If it is still insufficient, it iterates without exceeding the incremental upper limit to obtain the baseline camera parameter set that meets the hard constraint.

[0053] Residual composition optimization module: Using the baseline camera parameter set as the initial value, it refines the image by small perturbation steps with sensitivity decreasing from high to low, focusing on soft indicators such as the occlusion rate of people, the proportion of people in the frame, the visual center of gravity and the balance of white space. Each round only changes a single degree of freedom and verifies in real time that the integrity of landmarks does not decrease. When the gain is lower than the threshold or the iteration limit is reached, it stops and outputs the optimized camera parameter set and residual record.

[0054] The capture closed-loop fusion module performs consistency and stability checks on the benchmark solution and the optimized solution, and determines whether to capture based on the image shake amplitude, target tracking stability and focus sharpness threshold. After capturing, the panoramic channel is used to check the landmark integrity and the person occlusion rate and generate quality labels. The action sequence and step size selection are written into the experience cache, and the incremental upper limit, step size and priority of subsequent sensitivity estimation are updated to form an online closed loop.

[0055] In the motion sensitivity assessment module, person instance masks, landmark key part masks, and structural key point sets are extracted from panoramic and close-up images. Combining camera intrinsic and extrinsic parameters, single-step micro-explorations are performed on the horizontal sway angle, pitch angle, and focal length. The differences in composition score, landmark integrity, and person occlusion rate are calculated to obtain three-axis sensitivity. Based on this, an increment limit and step size are set to form a priority-ranked sequence of motion candidates. Specific content includes:

[0056] Image data acquired from a wide-angle camera and a controllable PTZ camera are denoted as multimodal perception data. The wide-angle camera provides panoramic image data for detecting and tracking tourist targets and obtaining the positional relationship of landmarks in the global scene. The PTZ camera provides local close-up image data for high-resolution acquisition of target tourists and landmark details, and panoramic perception of tourist distribution and relative positions of landmarks. The PTZ camera has a variable focal length lens, initially aligned with the direction of the landmark, and can adjust the viewing angle through panning, tilting, and zooming to acquire close-up images of tourists and landmarks.

[0057] An image composition evaluation function is established using a geometric calibration model of the camera and the scene. ,in , Indicates the horizontal and vertical rotation angles of the gimbal. Representing focal length, this function integrates indicators such as landmark coverage, person position and proportion, and aesthetic score. Using multimodal perception data, it can analyze the impact of each indicator on the current viewpoint. , The partial derivatives of the evaluation function are used to achieve differentiable optimization control of the agent. When some components of the evaluation function cannot be precisely differentiated, the agent model can automatically switch to zero-order optimization, that is, without gradient information, it adjusts through trial and error. The switching mechanism ensures that the agent can choose an appropriate optimization strategy under different evaluation metrics.

[0058] Specifically, the proxy model includes the following steps:

[0059] Environmental condition estimation: Based on the panoramic view provided by the wide-angle camera, the position of the tourist target in the world coordinate system is estimated. By analyzing the occlusion relationship between the target tourist and the landmark, as well as the boundary position of the landmark from the current pan-tilt-zoom (PTZ) viewpoint, the distance between the tourist and the camera can be further estimated using binocular and depth information. This provides scale information for subsequent calculations.

[0060] Image evaluation function It consists of several sub-evaluation items, including the landmark integrity evaluation. Character Composition Evaluation Overall aesthetic score This function accepts the current gimbal parameters. The system takes the scene state obtained from multimodal perception as input and outputs a comprehensive score that reflects the quality of the photo.

[0061] For the differentiable part, the gradient of the evaluation function with respect to each control parameter is calculated. This yields the parameter update direction for improving image quality; for parts where the gradient cannot be directly obtained, a zero-order strategy is used to evaluate the impact of small adjustments on the score, and the next action is determined by combining information from both aspects. , , .

[0062] The calculated control increment drives the camera to move or zoom in the specified direction, and the maximum step size for each action is set to ensure smooth convergence of the adjustment process.

[0063] Using the model described above, the photography agent acts as a virtual photographer. It understands the distribution of people and scenery within a scene and attempts to improve the composition quality of the photos by adjusting camera parameters. Before entering the formal optimization process, the agent first sets a starting camera direction based on the initial panoramic perception results, thus proceeding to the next step.

[0064] In the baseline viewpoint generation module, with a landmark integrity of no less than a threshold as a hard constraint, the pixel deviation of the landmark bounding box is converted into angle and field of view adjustment amounts in the order of horizontal swing angle, pitch angle, and focal length, and gradually corrected until the boundary is complete and falls within the safe area of ​​the image; if it is still insufficient, it iterates without exceeding the incremental upper limit to obtain a baseline camera parameter set that satisfies the hard constraint, the specific contents of which include:

[0065] Updated camera orientation and focal length This ensures that the landmarks meet the integrity constraint. At this point, the number of landmarks in the image must reach at least the threshold. The image is presented completely, and the target tourists are generally already inside the frame. The meaning is defined as follows:

[0066] Landmark integrity assessment Defined as the proportion of a visible landmark area or feature in the current image, relative to an ideal, complete landmark. For example, it can be determined by the detected landmark pixel area. Compared to the expected complete landmark area Calculation of the ratio: value range When the landmark is of an irregular shape, the key point detection method can also be used to calculate the completeness: This refers to the proportion of landmark feature points identified in the current view out of the total number of feature points.

[0067] Set a minimum required level of integrity, i.e., a landmark integrity threshold. ,For example This is an empirical value, meaning that a landmark's edge can be slightly cropped or extend beyond the field of view by no more than 5%. This threshold was determined through a large-scale survey of scenic area photos: generally, if a landmark is missing more than 5%, it will appear noticeably incomplete and cannot be considered a high-quality photo. Therefore... Using 0.95 ensures that the landmark is almost completely in the frame while leaving some room for the camera composition to accommodate the position of the people. This experience comes from photography manuals and user surveys. Generally, viewers are more tolerant of slight cropping of small parts such as the top and tip of the landmark, so 5% was chosen as the empirical threshold.

[0068] Landmark hard constraints are addressed using an iterative optimization or direct solution method. Since the size and location of landmarks are fixed and known in a wide-angle panorama, their coverage within the frame can be calculated based on the current gimbal angle. If... This indicates that the landmarks are incomplete and the camera needs to be adjusted. The adjustment should follow these strategies:

[0069] Horizontal Angle Adjustment: Check if the landmark is cropped at the horizontal edge of the image. If the left edge of the landmark extends beyond the viewfinder, adjust the pan / tilt head's horizontal angle. Rotate to the left by a certain angle If the right edge is cut, rotate to the right. . The number of pixels that can be offset from the edge of the landmark in the image. Calculated using camera perspective: ,in This is the current focal length. Using a pixel-to-angle scaling factor, calculate an approximate translation angle to move the landmark edge into the image. Re-evaluate after rotation. .

[0070] Vertical Angle Adjustment: Similarly, check the vertical integrity of the landmark. If the top of the landmark exceeds the frame, reduce the tilt angle. If the bottom is cut off, increase the pitch angle and adjust the angle. By vertical deviation pixels And internal parameter calculation: Post-adjustment assessment .

[0071] Focus adjustment: After satisfying the above horizontal and vertical angle adjustments, as follows Not yet reached Furthermore, the gimbal's field of view is already centered. This may be due to an excessively long focal length resulting in a narrow field of view. In this case, the field of view can be increased by reducing the focal length until... Or it may reach the lens's wide-angle limit. Zoom adjustment can calculate the required field-of-view expansion ratio based on the current L: Let... Preliminary estimate of the new focal length It is roughly assumed that the visible area of ​​a landmark is proportional to the field of view, and then fine-tuned. until The requirements are met.

[0072] The above adjustments are usually completed within one or two rounds because the landmark location is fixed and the target is clear. In the experimental environment, this can usually be achieved through one horizontal + vertical calibration and, if necessary, one zoom. The entire process is differentiably guided by the derivative of the landmark's edge with respect to the camera angle: since the wide-angle camera provides the landmark's position in the panorama, the camera rotation and the landmark's pixel offset are approximately linearly related. Therefore, the agent directly calculates the angle correction value to move the landmark's center into the center of the image, achieving rapid calibration. This demonstrates the role of a differentiable agent: given clear geometric constraints, it can directly calculate the correction action, equivalent to achieving the desired result in one step using gradients.

[0073] Parameter criteria: The criterion for completing Phase One is that the landmark integrity meets the standard: that is, when If the hard constraints of the landmarks are considered satisfied, the process moves to the next stage. If the constraints cannot be met within a finite number of steps, such as a limit of 3 adjustments, the method allows for further attempts. In practice, since the landmarks are fixed, the number of steps rarely exceeds 3. If the adjustment exceeds the predetermined number of steps and still fails to meet the requirements, it can be assumed that the initial positioning error is too large or that special environmental conditions have occurred. In this case, a degradation scheme will be triggered, such as relaxing the constraints. Slightly lower the requirements or suggest manual intervention. However, wide-angle perception guidance generally ensures that the initial pointing is good enough, making... Achieve the target quickly.

[0074] After completing Phase 1, output In addition to the landmark's boundary coordinates within the frame, since the landmark is already fully in the frame, the target tourist is usually also within the frame. At this point, the photo has met the basic requirements for usable snapshots: the main landscape is clear and complete, and the tourist is in the frame. However, simply meeting this requirement is not enough to make the photo marketable, because the person may be centered and obscure the landmark or be out of proportion, which means solving the composition optimization of the person and the landscape.

[0075] In the residual composition optimization module, using the baseline camera parameter set as initial values, the module refines the image by small perturbations based on soft metrics such as subject occlusion rate, subject occupancy ratio, visual center of gravity, and white space balance, decreasing sensitivity in each iteration. Each round changes only a single degree of freedom and verifies in real-time that landmark integrity does not decrease. The process stops when the gain falls below a threshold or reaches the iteration limit, outputting the optimized camera parameter set and residual records. Specific content includes:

[0076] By fine-tuning the camera's posture and zoom, the position and proportion of the subject are aesthetically optimized without compromising the landmark integrity achieved in Stage 1. Under this hard constraint of landmark integrity, softer indicators of harmony between people and scenery are further pursued, including ensuring the subject doesn't excessively obscure the landmark, that the subject is appropriately sized, and that the composition is balanced and aesthetically pleasing. This step aims to bridge the semantic gap between ordinary snapshots and professional photographs, significantly enhancing the image's visual appeal and shareability.

[0077] Combining image status and camera parameters And related assessment information includes:

[0078] Person detection and posture: the position of the target tourist in the image. and bounding box dimensions It can further obtain the outline or key point posture of the person, and at the same time identify whether there are other distracting parts of the tourist entering the picture and causing occlusion.

[0079] Landmark occlusion assessment: Identify whether target visitors are obscuring significant portions of a landmark. If a visitor's location overlaps with a landmark, identify the area of ​​the overlapping region or its importance weight.

[0080] Aesthetic rating Using a pre-trained image aesthetic evaluation model, an aesthetic score is given for the current image. The scoring system considers various visual factors, such as compositional balance, sharpness, and color, but composition plays a dominant role in this scenario. It's important to emphasize that this model is not a simple rule-based judgment, but rather one trained on a large amount of human preference data, enabling it to comprehensively assess the aesthetic appeal of a photograph.

[0081] When no pre-trained aesthetic model is loaded, a rule-based scoring system is automatically enabled. The input includes a single-frame image, a mask of the person instance, optional masks of key landmark parts, and the bounding rectangle of the person. The scoring range is 0 to 1, with higher values ​​indicating better composition. Scores include the following:

[0082] Center of gravity deviation score: Calculates the offset of the visual center of gravity of the person and the landmark as a whole relative to the center of the image, and normalizes it according to the image diagonal; the smaller the offset, the lower the score.

[0083] White space balance score: Calculate the distance from the outer rectangle of the figure to the left, right, top, and bottom edges of the image, determine the dispersion of the white space on the four sides and normalize it; the more balanced, the lower the score.

[0084] Percentage Deviation Score: Calculates the area percentage of the figure in the image and normalizes the deviation from the expected percentage (default 30%); the smaller the deviation, the lower the score.

[0085] Occlusion Score: When landmark masks are present, this score calculates the percentage of occlusion of key landmark areas by the character; the more occlusion, the higher the score; this score is 0 when there are no landmarks.

[0086] The four scores are each compressed to between 0 and 1, and the reverse value of the equally weighted average is taken as the aesthetic score. If business preferences need to be reflected, different weights can be set for the four scores, but the sum of the weights remains 1. When not labeled, they are weighted by default. The score is only used as a soft target and must meet the hard constraints at the same time: the landmark completeness is not less than 0.95. After meeting the hard constraints, the aesthetic score is not less than 0.70 as the selection threshold.

[0087] Ideal composition parameters: Some ideal values ​​set based on photographic knowledge or user preferences, such as:

[0088] Ideal Positioning of the Subject: A common compositional principle is to slightly offset from the center and to one side without obscuring the main part of the background. For example, the rule of thirds can be used to place the subject at points 1 and 3 horizontally, and the landmark at another point 1 and 3, to achieve balance. However, since both the subject and the landmark are main subjects, this example tends to favor misalignment rather than overlap. The positional deviation of the subject relative to the landmark can be calculated. , The goal is to make the figure's silhouette avoid key areas of the landmark. The ideal offset can be determined based on the size of the landmark, for example, placing the figure in the foreground on one side of the landmark.

[0089] Ideal proportions for people: refers to the appropriate size of a person in a photograph, expressed as a proportion deviation. This is used to measure the difference between the current figure size and the ideal size. The proportion of the figure's height in the image is defined as... , Photo height in pixels, ideal aspect ratio Based on experience, for example, a full-body photo typically shows the subject's height occupying about 1 / 2 to 1 / 3 of the image for a more aesthetically pleasing proportion. Any deviation from this proportion would be considered undesirable. ,when This indicates that the height of the figures is in line with expectations; The figures are too large, and the scenery is not well depicted. This indicates that the figures are too small or too far away to be easily identified.

[0090] ideal value The choice is based on travel photography experience: to balance the clarity of the person and the vastness of the scenery, the height of the person in the whole body should be around 30% [5]. If the person is just an embellishment of the scenery or you want to take a picture with a small person and a large scene, you can reduce the height of the person. If a prominent figure is desired, the elevation can be increased. However, for the sake of a general aesthetic that can be sold and shared, the elevation should be lowered. As an empirical value, it is derived from the statistical average of a large number of excellent travel photos.

[0091] Compositional stability: Consider negative space, headspace, etc. For example, leave a certain amount of space between the top of the person's head and the top edge of the photo. Avoid making it too high on the edge.

[0092] Image balance: If a person is positioned on one side of the frame, the background and scenery on the other side should visually balance the composition. This can be achieved through visual center of gravity calculation: assign weights to various elements of the image and calculate the distance of the center of gravity from the center. To minimize the deviation.

[0093] The camera parameters have been updated based on the above adjustments. This refers to the final camera orientation and focal length after a series of fine-tuning adjustments. Under these parameters, the photograph satisfies the following:

[0094] The landmark remains intact The figures do not obscure key parts of the landmarks, or the separation of figures from the scenery is achieved through misalignment; the size and proportion of the figures are close to ideal, with only slight deviations. Reduce to an acceptable range, for example That is, the difference between the figure's height and the ideal value should not exceed 10%; aesthetic score There is a significant improvement compared to Phase 1, namely, an increase in aesthetic scores. Reaching a certain threshold or above.

[0095] in, Defined as the increase in aesthetic score during the Phase 2 optimization process, for example... The score for entering Phase Two. For the optimized score, then This gain directly reflects the improvement in image quality from ordinary snapshots to optimized photos. Requirements Only a value significantly greater than 0 can prove the value of this step. According to internal experiments, the average... A score of 0.15 is enough for users to intuitively perceive a significant leap in photo quality; correspondingly, a score of 100 corresponds to an aesthetic score increase of 15 points. If... If the value is too small, that is, almost 0, it means that the original image is already near perfect or that the optimization has not had any effect.

[0096] The subject fine-tuning optimization employs an iterative adjustment strategy, simulating the photographer's repeated recomposition process. In each iteration, the agent selects a composition improvement action based on the current image evaluation. For example: 1. Avoiding landmark occlusion: If a subject is detected obscuring an important part of a landmark, the required direction for the subject to move away is calculated. For instance, if the subject is standing in the center of the landmark, the camera can be slightly panned to the side to move the subject to a position with a cleaner background. (Rotation angle...) Calculate based on the proportion of overlap between the figure and the landmark in the image. The zero-order method involves attempting small left or right tilts of the angle. For example, when shooting at a 1° angle, compare which side has less occlusion and a higher score, and then continue moving in that direction until the occlusion is reduced to a minimum acceptable level. Since the relative positions of the person and the landmark are known in the wide-angle view, it can generally be determined whether moving the person to the left or right of the landmark is better. For example, if the person was originally slightly to the right and in front of the landmark and was obstructing it, then turn the camera slightly to the right to move the person to the left of the frame relative to the landmark, revealing more of the landmark. The entire process is constrained by the requirement that the landmark not go out of the frame, so the translation range is monitored simultaneously. Preventing from falling .

[0097] Adjust character proportions: If The figure is too large, indicating that the figure size is inappropriate. (Regarding...) It allows you to zoom in to enlarge the subject while keeping landmarks within the frame; [This is for...] If a person is too large and fills the entire frame, you can zoom out or move the camera back within acceptable limits. If the camera position is fixed and cannot be moved back, you can only increase the sense of distance between the person and the scene by reducing the focal length. Utilize perspective... ,like If approximately invariant, then Therefore, a new focal length can be selected. As a first step, estimate the height, then take photos to verify the figure's height, and make fine adjustments. Until satisfied Range. At the same time, the integrity of the landmark is continuously monitored. If the edge of the landmark is about to go out of the frame due to the zooming in of the figure, it is slightly shifted to correct or minor defects are accepted.

[0098] Compositional Balance and Aesthetic Enhancement: After ensuring the approximate proportions of landmarks and figures are appropriate, further consider the overall compositional aesthetics. For example, apply compositional rules such as the rule of thirds and diagonals: place figures near intersections, distribute landmarks along diagonals, etc. The proxy can be adjusted using aesthetic scoring gradients: Since the aesthetic scoring model is a black box and gradients cannot be directly analyzed, zero-order probing can be used. Multiple perturbations can be tried near the current parameters, and the scores after these attempts can be calculated. Choose the adjustment that will provide the greatest improvement. If multiple attempts fail to improve the score, it indicates that the score has reached a plateau. Common optimizations include:

[0099] White space and edge spacing: Ensure adequate space above the character's head. The figures should not be flush with the border on the left and right sides, nor should the top of the landmarks be flush with the border. The harmonious spacing between each subject and the border is found through fine-tuning the pan and tilt. This is usually reflected in the aesthetic score; excessively cramped spacing will result in a deduction of points.

[0100] Reduce cluttered backgrounds: If the aesthetic score is low and other tourists are observed interfering with the subject's background, wait for the distractions to move away or slightly adjust the angle to avoid them. For example, if other tourists are detected walking past the target, pause taking the photo or adjust the angle to avoid them. This waiting or adjustment process is also considered an optimization action. In practice, an occlusion tolerance threshold can be set. If the occlusion percentage exceeds this value, shooting will not be triggered. This value is empirically set at 0.2, allowing a maximum of 20% of the area to be obscured by minor visitors. This is based on subjective surveys: most people can accept a few passersby in the corners of a photo, but not if they block the main scenery or subject. If the occlusion exceeds the threshold, shooting convergence will be temporarily suspended until the obstructing party leaves or attempts are made to change the angle to reduce the occlusion.

[0101] The optimization process typically requires multiple small adjustments. Set an upper limit on the number of steps in the action. This limits the number of iterations to prevent excessive attempts from causing missed shooting opportunities or tourists to leave. The experience value can be set according to the needs of the site, for example... This indicates a maximum of 10 fine-tuning actions. This value depends on the device's response speed and the visitor's movement speed: with a gimbal camera in use, adjustments up to 10 times can be completed within 2 to 3 seconds. Most visitors don't change position much within a few seconds of being photographed, so 10 steps is sufficient for convergence without causing delays. If more than... If the score is still not satisfactory, I will choose the best composition at this point and shoot directly to avoid missing the opportunity. The value of was determined through multiple field tests: when the number of adjustment steps was less than 5, it sometimes could not be fully optimized, and when it exceeded 15 steps, there was almost no additional improvement and it was a waste of time. Finally, 10 was selected as a compromise.

[0102] Aesthetic score gain Using improvements in other metrics as criteria, optimization is considered convergent and the iteration can end when any of the following conditions are met:

[0103] The aesthetic score improvement in two consecutive iterations was insufficient to reach the minimum gain threshold. ,Right now . Take 0.01, which is about 1% improvement or 1 point on the corresponding rating scale. This value is determined based on the difference that the human eye can perceive: generally, when the rating improvement is less than 1%, people can hardly feel the change, and further adjustments will not yield much benefit.

[0104] The preset maximum number of steps has been reached. Furthermore, the current composition already meets all the basic requirements. , Small, with no obvious obstructions, even if the score still has a slight room for improvement, we stop to ensure efficiency and trust that the current composition is already optimal;

[0105] The aesthetic score reaches an absolute high value, approaching the model's upper limit, for example... This indicates that the photo is already excellent and requires no further optimization.

[0106] The target tourist began to move, causing the composition to quickly deviate.

[0107] Upon completion of Phase Two, the optimized camera parameters are obtained. At this point, the image should be marketable: landmarks and tourists should be clear, the composition balanced, and the aesthetics pleasing. Record the final aesthetic score at this point. And the improvement compared to stage one This serves as a basis for performance evaluation. In common cases, If all values ​​are positive, it proves the optimization is effective; if unexpected results occur... Roll back to the previous parameter scheme with a higher score to avoid outputting a suboptimal composition.

[0108] In the capture-closed-loop fusion module, consistency and stability checks are performed on the baseline solution and the optimized solution. Whether to capture the image is determined by the image shake amplitude, target tracking stability, and focus sharpness threshold. After capture, the panoramic channel is used to verify the landmark integrity and human occlusion rate and generate quality labels. The action sequence and step size selection are written into the experience cache, updating the incremental upper limit, step size, and priority of subsequent sensitivity estimations, forming an online closed loop. Specific content includes:

[0109] Combining the state after the last iteration, including the current parameters of the PTZ camera. Aesthetic rating Number of iterations And related indicators.

[0110] The capture trigger signal and the final photo: The capture signal drives the camera to press the shutter under the current parameters to take a high-resolution photo and store the image. The main determination conditions include:

[0111] Aesthetic score convergence: As mentioned above, when the increase in aesthetic score is below a threshold... If there is no significant improvement after several consecutive steps, it is considered convergence. If the score has reached a high level at this point, it is deemed acceptable to shoot. Specifically, it can be requested that the final step be... Adjustments are underway, such as , Improvement To avoid stopping after a single fluctuation.

[0112] Action steps exhausted: If the iteration has reached The upper limit is also considered as convergence. In principle, shooting should be done in a timely manner, and greedy optimization should no longer be pursued.

[0113] Environmental stability: Shooting is only performed when the target tourist has stabilized shortly after the most recent adjustment and there are no sudden obstructions. This prevents taking undesirable photos even when the tourist has just moved away or blocked the frame after the composition has been adjusted. Target displacement is detected over multiple consecutive frames; if it is less than a set threshold, the frame is considered stable. If the target begins to move away, early shooting or abandonment may be triggered. If someone suddenly passes by an obstructing landmark or tourist, the photographer can wait briefly for the obstructor to move before shooting.

[0114] Landmark Integrity Preservation: Ensure landmarks remain intact at the final shooting moment. If any action during final fine-tuning results in... Slightly lower It should be corrected or at least confirmed that it will not affect the main effect before shooting. Phase two is usually already maintained. However, in the pursuit of aesthetics The decline is close to Boundaries, requirements before shooting Not less than , A tolerance of 0.02 can be used as a small concession. If it exceeds this tolerance, it needs to be finely adjusted and corrected before taking the picture again.

[0115] When the above conditions are met or the overall effect is satisfactory, the composition optimization process is considered complete. At this point, the photography agent will output the final action: including maintaining the current gimbal parameters and triggering the shutter.

[0116] After the convergence determination is passed, a shooting command is sent to the camera. Since the gimbal camera has stabilized in the optimized position, the photo acquired at this moment should meet the set high-quality standards: the landmark is clear and complete, the person is appropriately positioned, and the overall photo is aesthetically pleasing. At the same time, a wide-angle camera can be used to confirm again at the moment of capture that no one is obstructing the target or landmark. Once the photo is taken, further processing will be carried out, such as comparing it with other candidate photos to select the best one, which is not the focus of this embodiment.

[0117] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0118] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0119] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and inventive constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0120] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0121] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0122] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal perception-based non-intrusive capture and intelligent selection system for tourists in scenic areas, characterized in that: The system includes: a motion sensitivity assessment module, a baseline viewpoint generation module, a residual composition optimization module, and a capture closed-loop fusion module, with signal connections between the modules; Action sensitivity assessment module: Extracts person instance masks, landmark key part masks, and structural key point sets from panoramic and close-up images; Combines camera intrinsic and extrinsic parameters, performs single-step micro-exploration of horizontal swing angle, pitch angle, and focal length respectively, calculates the difference in composition score, landmark integrity, and person occlusion rate, and obtains three-axis sensitivity; Based on this, sets the increment upper limit and step size to form a priority-arranged action candidate sequence; The reference view generation module uses the landmark integrity of not less than the landmark integrity threshold as a hard constraint. It converts the pixel deviation of the landmark outer frame into angle and field of view adjustment amount in the order of horizontal swing angle, pitch angle and focal length, and gradually corrects it until the boundary is complete and falls into the safe area of ​​the screen. If the hard constraint is still not met, it iterates without exceeding the incremental upper limit to obtain the reference camera parameter set that meets the hard constraint. The residual composition optimization module uses the baseline camera parameter set as initial values. It refines the parameters by small-step perturbation based on four soft indices: occlusion rate of the subject, proportion of the subject in the frame, visual center of gravity deviation, and white space on all four sides of the frame. Each round only changes a single degree of freedom and verifies in real-time that landmark integrity does not decrease. The module stops when the gain falls below a threshold or reaches the iteration limit, and outputs the optimized camera parameter set and residual records. Specifically, it calculates the four soft indices (occlusion rate of the subject, proportion of the subject in the frame, visual center of gravity deviation, and white space on all four sides of the frame) using the baseline camera parameter set as initial values. It selects a single degree of freedom based on its sensitivity from high to low, applies small-step perturbation to obtain candidate solutions, and checks the landmark integrity of the candidate solutions to ensure it is not lower than a landmark integrity threshold. If this is not met, it backs down the perturbation and retry with a smaller step size. It records the residual and soft index gain for each round. When the soft index gain falls below the convergence threshold or reaches the iteration limit for a preset number of consecutive iterations, it outputs the optimized camera parameter set and residual records. The capture closed-loop fusion module performs consistency and stability checks on the baseline and optimized solutions, determining whether to capture based on image stability, target tracking stability, and focus sharpness and their corresponding thresholds. After capture, the panoramic channel is used to verify landmark integrity and human occlusion rate and generate quality labels. The action sequence and step size selection are written into the experience cache, and the incremental upper limit, step size, and priority of subsequent sensitivity estimation are updated to form an online closed loop. Specifically, this includes: calculating the difference vector between the baseline camera parameter set and the optimized camera parameter set in the three degrees of freedom of horizontal swing angle, pitch angle, and focal length, and comparing it with the consistency threshold. If any component exceeds the limit, backtracking is performed along the direction with the smaller difference and soft indicators are rechecked. The target is calculated within a short time window. The system detects displacement and image shake amplitude, and checks focus sharpness. If the displacement is less than a set threshold, the image shake amplitude is lower than the shake threshold, and the focus sharpness is higher than the sharpness threshold, the system is considered stable. At the moment of capture, the system checks the landmark integrity. If the landmark integrity threshold boundary is approached, the final parameters are slightly corrected before the capture is triggered. The system calculates the landmark integrity and person occlusion rate of the final image using the panoramic channel and generates a quality label. The consistency check result, stability check result, actual action sequence and step size, and soft and hard index response are written into the experience cache. Based on the experience cache, the sensitivity statistics are updated online, and the incremental upper limit, step size, and action candidate sequence priority are adjusted accordingly for subsequent use.

2. The multimodal perception-based non-intrusive capture and intelligent selection system for scenic area tourists according to claim 1, characterized in that, The motion sensitivity assessment module is specifically used to: read camera intrinsic and extrinsic parameters and calculate image elements using person instance masks, landmark key part masks and structural key point sets; By applying positive and negative single-step micro-explorations to the three degrees of freedom of horizontal swing angle, pitch angle, and focal length, the changes in composition score, landmark integrity, and human occlusion rate were obtained. Normalize the three-degree-of-freedom changes to form a sensitivity vector, generate and output the action candidate sequences containing degrees of freedom, step size and upper limit of increment in descending order of sensitivity.

3. The non-intrusive capture and intelligent selection system for scenic area tourists based on multimodal perception as described in claim 1, characterized in that, The reference viewpoint generation module is specifically used to: convert the horizontal and vertical pixel deviations of the landmark outer frame relative to the center of the image into corresponding angle changes and field of view changes; According to the priority of the action candidate sequence, horizontal sway angle correction, pitch angle correction and focal length correction are applied in sequence. After each correction, the landmark integrity is calculated and compared with the landmark integrity threshold. If the landmark integrity threshold is not reached, the iteration continues without exceeding the increment limit and step size. When the landmark integrity threshold is reached and the landmark boundary is within the safe zone, the parameters are frozen and the reference camera parameter set is output.

4. The non-intrusive capture and intelligent selection system for scenic area tourists based on multimodal perception as described in claim 1, characterized in that, The occlusion rate of a person is calculated as the ratio of the intersection area of ​​the person instance mask and the landmark key part mask to the area of ​​the landmark key part mask. In the residual optimization process, priority is given to perturbing the degree of freedom direction that makes the occlusion rate of the character monotonically decrease. When occlusion rate oscillations occur, reduce the step size and maintain landmark integrity no lower than the landmark integrity threshold.

5. The non-intrusive capture and intelligent selection system for tourists in scenic areas based on multimodal perception as described in claim 1, characterized in that, The proportion of the figure in the frame is calculated as the ratio of the figure's outer frame area to the total frame area, and then compared with the expected proportion. Based on the camera's current field of view, the proportional deviation is converted into a focal length change and proportional correction is performed. After each proportional correction, the landmark integrity is evaluated simultaneously. If it falls below the landmark integrity threshold, the proportional correction for that round is canceled, and the focal length change is reduced and retried under the condition that it does not exceed the step size and increment limit given by the action candidate sequence. When the scaling correction causes the visual center of gravity deviation of the image to increase compared to before the correction and exceeds the preset visual center of gravity deviation threshold, a small correction of the horizontal swing angle or pitch angle is performed without reducing the integrity of the landmark. The correction range is constrained by the aforementioned step size and increment limit.

6. The non-intrusive capture and intelligent selection system for tourists in scenic areas based on multimodal perception as described in claim 1, characterized in that, Calculate the visual center of gravity for the main character area and the landmark area respectively, and synthesize the visual center of gravity of the image according to the weights. Calculate the deviation of the center of gravity from the center of the image and the white space on the four sides of the image. Perform small-step perturbations in the direction of perturbations that simultaneously reduce center of gravity deviation and improve white space balance; When approaching the landmark integrity threshold boundary, reduce the step size until the safety margin is restored and record the corresponding residual.

7. The non-intrusive capture and intelligent selection system for scenic area tourists based on multimodal perception as described in claim 1, characterized in that, When the optimized camera parameter set and the baseline camera parameter set exceed the limits in the consistency check and the stability is insufficient, the baseline camera parameter set or the most recently stable intermediate parameter set should be selected to perform the snapshot. The gimbal and zoom lens are driven to perform minute angle and focal length movements according to the selected parameter group; Before and after the action, assess the integrity of the landmark, the occlusion rate of the person, the proportion of the person in the frame, and the white space of the visual center of gravity, and record the execution log.

Images