Intelligent shooting guiding method and device, equipment and storage medium

Abstract

The invention relates to an intelligent shooting guiding method and device, equipment and a storage medium, and the method comprises the steps: obtaining a target image containing target semantic information of an image and an image depth map based on a deep learning model and a depth estimation model, obtaining a feature point set of the target image, and generating an initial global map based on the feature point set, generating a semantic tag map based on the target semantic information, the image depth map and the initial global map; performing feature point extraction and screening on a target image, calculating semantic weight values of a plurality of obtained first feature points, determining a moving path based on the semantic weight values and a path planning algorithm, and generating a control instruction including a linear velocity and an angular velocity for controlling the shooting equipment to move, and obtaining and obtaining a detection quantity based on environment change in real time; dynamically adjusting the moving path, and obtaining and controlling the shooting equipment to move based on the dynamically adjusted moving path; intelligent shooting guiding can be realized, and adaptability and flexibility of path planning in the shooting guiding process are enhanced.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to an intelligent shooting guidance method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of drones, mobile devices, and intelligent photography technologies, demand for their application in architectural surveying and mapping, cultural heritage preservation, virtual reality, and other fields is growing. To achieve high-quality photography, photography guidance technology has become a key link, involving core technologies such as path planning, environmental perception, and dynamic adjustment.

[0003] Traditional shooting guidance methods mainly rely on preset paths or manual intervention, and are difficult to adapt to complex dynamic environments. For example, in cultural relics protection scenarios, shooting paths need to be frequently adjusted to capture details, but preset paths cannot cope with changes in occlusion or obstacles. Manual intervention is inefficient and easily affected by subjective factors.

[0004] In addition, although existing SLAM technology can achieve positioning and map construction, it lacks semantic understanding capabilities and cannot effectively combine target recognition information; although deep learning algorithms can process image semantics, they have not been deeply integrated with SLAM technology, resulting in insufficient path planning and real-time adjustment capabilities in dynamic scenes. Summary of the Invention

[0005] The present application provides an intelligent shooting guidance method, device, equipment and storage medium, which can realize intelligent shooting guidance and enhance the adaptability and flexibility of path planning during the shooting guidance process.

[0006] In the first aspect, the present application provides an intelligent shooting guidance method, including: performing target detection and semantic segmentation processing on an input image based on a deep learning model to obtain a target image containing target semantic information; generating an image depth map corresponding to the target image based on a depth estimation model, performing feature point extraction and feature point matching processing on the target image to obtain a feature point set corresponding to the target image, generating an initial global map based on the feature point set, and generating a semantic label map based on the target semantic information, the image depth map and the initial global map; performing feature point extraction and feature point screening processing on the target image to obtain multiple first feature points, and calculating the semantic weight values corresponding to each of the multiple first feature points, determining a moving path based on the semantic weight values and a preset path planning algorithm; controlling the shooting device to move based on the moving path, and obtaining an environmental change detection amount in real time during the movement, dynamically adjusting the moving path based on the environmental change detection amount to obtain a dynamically adjusted moving path, and controlling the shooting device to move based on the dynamically adjusted moving path.

[0007] In one possible implementation, target detection and semantic segmentation processing are performed on an input image based on a deep learning model to obtain a target image containing target semantic information, specifically including: the deep learning model includes a target detection model and a semantic segmentation model; an image is acquired in real time, the image is preprocessed to obtain a first preprocessed image, the first preprocessed image is input into the target detection model, so that the target detection model extracts features from the first preprocessed image to obtain an image feature map, the image feature map is grid-divided to obtain multiple feature grids, and target detection processing is performed on the multiple feature grids respectively to obtain target classes corresponding to the target. The method comprises the following steps: first, performing pixel-level classification processing on the fused image, obtaining a category label corresponding to each pixel point, generating a semantic segmentation category label map, performing semantic information extraction on the semantic segmentation category label map, extracting the target category and target boundary information corresponding to each target, and generating a target image containing target semantic information; wherein, the target semantic information includes the target category and the target boundary information corresponding to each target.

[0008] In a possible implementation, feature point extraction and feature point matching are performed on the target image to obtain a feature point set corresponding to the target image, specifically including: performing corner point detection on the target image based on the ORB algorithm to obtain multiple corner points in the target image, and respectively obtaining corner point directions and corner point descriptors corresponding to each of the multiple corner points; integrating the corner point directions and corner point descriptors corresponding to each of the multiple corner points to generate an initial feature point set; screening the initial feature point set based on the target semantic information in the target image to obtain an optimized feature point set; obtaining adjacent frame images corresponding to the target image, and obtaining adjacent frame feature point sets corresponding to the adjacent frame images; calculating the feature point distance between each optimized feature point in the optimized feature point set and each adjacent frame feature point in the adjacent frame feature point set, and determining a set of matching feature point pairs between the optimized feature point set and the adjacent frame feature point set based on the feature point distance; extracting target matching feature points in the target image from the matching feature point pair set to obtain a feature point set corresponding to the target image.

[0009] In one possible implementation, an initial global map is generated based on the feature point set, and a semantic label map is generated based on the target semantic information, the image depth map, and the initial global map. Specifically, the method includes: obtaining multiple associated images corresponding to the target image, and obtaining associated feature point sets corresponding to each of the multiple associated images; performing feature point matching processing on the feature point set corresponding to the target image with the associated feature point sets corresponding to each of the multiple associated images to obtain multiple associated feature point matching point pairs, determining a posture transformation of the shooting device based on the multiple associated feature point matching point pairs, and constructing an initial global map based on the posture transformation and the multiple associated feature point matching point pairs; determining the two-dimensional coordinates of the target based on the target semantic information, determining the depth information of the target based on the image depth map, determining the target position of the target based on the two-dimensional coordinates and the depth information, and marking the target on the initial global map based on the target position to update the initial global map; and setting a target category label for the target position corresponding to the target on the updated initial global map based on the target semantic information to generate a semantic label map.

[0010] In one possible implementation, the semantic weight values corresponding to each of the multiple first feature points are calculated, and the moving path is determined based on the semantic weight values and a preset path planning algorithm, specifically including: respectively obtaining the target categories corresponding to each of the multiple first feature points, and determining the target category weight values corresponding to the multiple first feature points based on the target categories, and respectively calculating the distances between the multiple first feature points and the target to be photographed, and determining the semantic weight values corresponding to the multiple first feature points based on the target category weight values and the distances; obtaining the path starting point and path end point of the path to be generated, searching for the target first feature point between the path starting point and the path end point based on a preset path planning algorithm, and generating an initial moving path based on the path starting point, the path end point and the target first feature point, and calculating the first path total cost corresponding to the initial moving path based on a path cost function; adjusting the path cost function based on the target semantic information, and optimizing the initial moving path based on the adjusted path cost function to obtain an optimized moving path.

[0011] In one possible implementation, generating an image depth map corresponding to the target image based on a depth estimation model specifically includes: performing image preprocessing on the target image to obtain a second preprocessed image; inputting the second preprocessed image into the depth estimation model so that the depth estimation model predicts a first distance from each pixel in the second preprocessed image to the shooting device, and generating a first image depth map based on the first distance; obtaining multiple frames of continuous images corresponding to the target image, and obtaining the first continuous image depth map corresponding to the multiple frames of continuous images; fusing the first continuous image depth map and the first image depth map to obtain a first fused depth map, and filtering the first fused depth map to obtain an image depth map corresponding to the target image.

[0012] In one possible implementation, the intelligent guidance method provided by the present application further includes: when it is detected that the shooting device has a shooting device deviation, calculating the current field of view coverage of the shooting device based on a preset field of view coverage evaluation formula; when the current field of view coverage is less than a preset field of view coverage threshold, calculating the offset angle between the target area center of the target and the field of view center of the shooting device, calculating the adjusted offset angle based on the offset angle, and generating an angle adjustment instruction based on the adjusted offset angle, so that the shooting device is controlled to adjust the angle based on the angle adjustment instruction.

[0013] In the second aspect, the present application provides an intelligent shooting guidance device, including: a semantic segmentation module, a semantic label map generation module, a moving path generation module and a shooting equipment movement control module; wherein, the semantic segmentation module is used to perform target detection and semantic segmentation processing on the input image based on the deep learning model to obtain a target image containing target semantic information; the semantic label map generation module is used to generate an image depth map corresponding to the target image based on the depth estimation model, perform feature point extraction and feature point matching processing on the target image to obtain a feature point set corresponding to the target image, generate an initial global map based on the feature point set, and generate a target image based on the target semantic information, the target image and the semantic label map generation module. The image depth map and the initial global map are used to generate a semantic label map; the moving path generation module is used to extract and screen the feature points of the target image to obtain a plurality of first feature points, and calculate the semantic weight values corresponding to each of the plurality of first feature points, and determine the moving path based on the semantic weight values and a preset path planning algorithm; the shooting device movement control module is used to control the shooting device to move based on the moving path, and obtain the environmental change detection amount in real time during the movement, dynamically adjust the moving path based on the environmental change detection amount, obtain the dynamically adjusted moving path, and control the shooting device to move based on the dynamically adjusted moving path.

[0014] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.

[0016] The present invention provides a method that has the following advantages over the prior art:

[0017] The method comprises the following steps: performing target detection and semantic segmentation on an input image based on a deep learning model to obtain a target image containing target semantic information; generating an image depth map corresponding to the target image based on a depth estimation model, performing feature point extraction and feature point matching on the target image to obtain a feature point set corresponding to the target image, generating an initial global map based on the feature point set, and generating a semantic label map based on the target semantic information, the image depth map, and the initial global map; performing feature point extraction and feature point screening on the target image to obtain a plurality of first feature points, and calculating semantic weight values corresponding to each of the plurality of first feature points, and determining a movement path based on the semantic weight values and a preset path planning algorithm; controlling a shooting device to move based on the movement path, and obtaining an environmental change detection amount in real time during the movement, dynamically adjusting the movement path based on the environmental change detection amount to obtain a dynamically adjusted movement path, and controlling the shooting device to move based on the dynamically adjusted movement path. Compared with the prior art, the technical solution of the present application realizes intelligent planning and dynamic adjustment of the shooting path through the deep integration of deep learning and SLAM technology, significantly improving shooting efficiency, adaptability, and intelligence level. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0021] Figure 1 This is a flowchart of an embodiment of an intelligent shooting guidance method provided by the present application;

[0022] Figure 2 This is a structural diagram of an embodiment of an intelligent shooting guidance device provided by the present application;

[0023] Figure 3 This is a structural diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0024] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] The disclosure below provides many different embodiments or examples for implementing different structures of the present application. In order to simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, these are merely examples and are not intended to limit the present application. In addition, the present application may repeat reference numbers and / or letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0026] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0027] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0028] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0030] Example 1, see Figure 1 , Figure 1 This is a flow chart of an embodiment of an intelligent shooting guidance method provided by the present application. Figure 1 As shown, the method includes steps 101 to 104, which are specifically as follows:

[0031] Step 101: Perform target detection and semantic segmentation on the input image based on the deep learning model to obtain a target image containing target semantic information.

[0032] In one embodiment, the deep learning model includes a target detection model and a semantic segmentation model.

[0033] Specifically, the target detection model includes but is not limited to the YOLO-Mask or MMDetection3D model; the semantic segmentation model includes but is not limited to a network architecture such as a fully convolutional neural network (FCN) or U-Net.

[0034] In one embodiment, an image is acquired in real time, and the image is preprocessed to obtain a first preprocessed image.

[0035] Specifically, the image is obtained from a data stream captured in real time by a shooting device.

[0036] Specifically, the preprocessing of the image includes but is not limited to image normalization, image size adjustment, and image enhancement.

[0037] Specifically, the image is resized so that the image is scaled to the fixed output size of the deep learning model, and the scaled image is normalized to obtain a normalized image. The normalized image is enhanced using image enhancement techniques such as random cropping, rotation, and brightness adjustment to obtain a first preprocessed image, so as to improve the robustness of the model to different scenes and environments and reduce the risk of overfitting.

[0038] In one embodiment, the first preprocessed image is input into the target detection model so that the target detection model performs feature extraction on the first preprocessed image to obtain an image feature map, performs grid division processing on the image feature map to obtain multiple feature grids, and performs target detection processing on the multiple feature grids respectively to obtain the target category, target bounding box and target mask corresponding to the target, and performs fusion processing on the preprocessed image based on the target category, the target bounding box and the target mask to obtain a fused image.

[0039] Specifically, target detection models such as YOLO and MMDetection3D are used to extract features of the preprocessed image through a convolutional neural network (CNN). The network structure usually contains multiple convolutional layers and pooling layers, which are used to extract image features layer by layer to obtain an image feature map.

[0040] Specifically, based on feature extraction, the model simultaneously predicts the category and bounding box of objects in the image through regression. For example, the YOLO-Mask model divides the image into multiple grids, each grid is responsible for detecting targets in a specific area, and outputs the target category and the target bounding box (x, y, w, h) of the target. (x, y) is the coordinate of the target center point, and w and h are the width and height. In addition, based on target detection, the YOLO-Mask model generates a pixel-level mask for each target through an additional segmentation branch. Within the target bounding box, a structure similar to Mask R-CNN is used to perform fine segmentation on the target area to generate a target mask.

[0041] Specifically, the non-maximum suppression method is used to filter the detected bounding boxes and remove redundant boxes with high overlap; by calculating the intersection over union (IoU), the detection results with high confidence and no duplication are retained to improve the detection accuracy.

[0042] Specifically, the target category, the target bounding box, and the target mask are marked on the preprocessed image to obtain a fused image.

[0043] In one embodiment, the fused image is input into the semantic segmentation model so that the semantic segmentation model performs pixel-level classification processing on the fused image to obtain the category label corresponding to each pixel point, generate a semantic segmentation category label map, and perform semantic information extraction on the semantic segmentation category label map to extract the target category and target boundary information corresponding to each target, and generate a target image containing target semantic information; wherein, the target semantic information includes the target category and the target boundary information corresponding to each target.

[0044] Specifically, a semantic segmentation model such as a fully convolutional neural network (FCN) or U-Net is used to classify the preprocessed image pixel by pixel, and a semantic segmentation model such as a fully convolutional neural network (FCN) or U-Net is used to classify the preprocessed image pixel by pixel, and output the category label c of each pixel. i , and mark each pixel as a preset category to form a semantic segmentation category label map.

[0045] Specifically, the target category, bounding box coordinates, mask information, and category label are extracted from the target detection results and integrated to generate a target image containing target semantic information. The target image containing target semantic information can be expressed as:

[0046] R={(x i ,y i ,w i ,h i ,c i ,mask i )|i=1,…,n};

[0047] Among them, (x i ,y i ) is the target center coordinate, w i ,h i is the width and height, c i For category, mask i is a pixel-level mask.

[0048] Step 102: Generate an image depth map corresponding to the target image based on the depth estimation model, perform feature point extraction and feature point matching on the target image to obtain a feature point set corresponding to the target image, generate an initial global map based on the feature point set, and generate a semantic label map based on the target semantic information, the image depth map and the initial global map.

[0049] In one embodiment, the target image is preprocessed to obtain a second preprocessed image; the second preprocessed image is input into a depth estimation model so that the depth estimation model predicts a first distance from each pixel in the second preprocessed image to the shooting device, and generates a first image depth map based on the first distance.

[0050] Specifically, a target image is obtained and used as the input of the depth estimation model. Therefore, in order to ensure that it can adapt to the requirements of the depth estimation model, the target image needs to be processed; for example, the size of the target image is adjusted and converted into the input size specified by the depth estimation model to ensure that the depth estimation model can process the image data normally; at the same time, the brightness of the target image is standardized to eliminate the brightness differences caused by different lighting conditions, so that the target image has consistency in brightness, thereby providing stable input for subsequent depth estimation.

[0051] Specifically, the depth estimation model includes but is not limited to Monodepth2, wherein Monodepth2 is a depth estimation model built based on a convolutional neural network (CNN), the core principle of which is to learn the pixel-level depth information of a single image to predict the distance of each pixel to the camera.

[0052] Specifically, when the depth estimation model predicts the first distance from each pixel in the second preprocessed image to the shooting device, it adopts an unsupervised learning approach to perform deep learning by comparing the left and right disparities of the target image or based on the temporal consistency in the video stream; the model processes the input target image and finally generates a first image depth map; in this first image depth map, the depth value of each pixel represents the relative distance from the pixel to the shooting device, and the depth information of different areas in the target image is converted into a two-dimensional first image depth map, which can reflect the relative position of the object in space.

[0053] In one embodiment, multiple frames of continuous images corresponding to the target image are obtained, and the first continuous image depth map corresponding to the multiple frames of continuous images is obtained; the first continuous image depth map and the first image depth map are fused to obtain a first fused depth map, and the first fused depth map is filtered to obtain an image depth map corresponding to the target image.

[0054] Specifically, in order to improve the accuracy of depth estimation, multiple frames of continuous images corresponding to the target image are usually obtained, and the system inputs the multiple frames of continuous images into the depth estimation model based on the above steps to obtain the first continuous image depth map corresponding to the multiple frames of continuous images.

[0055] Specifically, the first continuous image depth maps corresponding to the multiple frames of continuous images are fused with the first image depth map of the target image. By combining the depth information of each frame, noise and inaccuracy that may exist in single-frame estimation can be eliminated.

[0056] Specifically, when the first continuous image depth map and the first image depth map are fused, a fused depth value corresponding to each pixel in the first image depth map is calculated based on a preset fusion calculation formula, and a first fused depth map is obtained based on the fused depth value.

[0057] Specifically, the preset fusion calculation formula is as follows:

[0058] D o pt(p)=(1 / k)ΣD i (p),i=1tok

[0059] Where k is the number of sliding window frames, D i (p) represents the depth value of pixel p in image frame i,

[0060] D o pt(p) is the fusion depth value corresponding to pixel p.

[0061] Specifically, after the fusion is completed, a filtering algorithm, such as mean filtering or Gaussian filtering, is used to optimize the first fused depth map to eliminate local instability and enhance the smoothness and accuracy of the first fused depth map. After these processes, an image depth map corresponding to the target image is finally obtained.

[0062] In one embodiment, corner point detection is performed on the target image based on the ORB algorithm to obtain multiple corner points in the target image, and the corner point directions and corner point descriptors corresponding to each of the multiple corner points are obtained respectively; the corner point directions and corner point descriptors corresponding to each of the multiple corner points are integrated to generate an initial feature point set; based on the target semantic information in the target image, the initial feature point set is screened to obtain an optimized feature point set.

[0063] Specifically, in order to improve the stability of ORB feature point extraction, the target image is first preprocessed, wherein the image preprocessing includes but is not limited to image size adjustment, color standardization, etc.

[0064] Specifically, the preprocessed target image is input into the ORB extraction algorithm. The ORB algorithm first uses the FAST corner detection algorithm to detect significant corners in the image. The FAST algorithm compares the brightness values of a pixel with multiple pixels in its neighborhood. If the brightness difference between the pixel and a sufficient number of pixels in its neighborhood is large, it is judged as a corner point. The corner point direction is determined by calculating the grayscale centroid in the corner point neighborhood to enhance the robustness of the feature point to rotation. At the same time, a BRIEF descriptor is generated for each corner point. The BRIEF descriptor is a binary string generated by comparing the brightness values of pixel pairs at different positions in the corner point neighborhood.

[0065] Specifically, the corner point directions and corner point descriptors corresponding to the multiple corner points are integrated to generate an initial feature point set P = {p i |i=1,...,m}, where each p i Contains the position, orientation and BRIEF descriptors of the corner points.

[0066] Specifically, the initial feature point set is screened based on the target semantic information in the target image. For example, the target category and boundary information in the target image are combined to determine whether each feature point is located in an important area. If so, the feature points in the important area are prioritized. The distance between feature points and the similarity of their descriptors are calculated to determine whether feature points are duplicated. If two feature points are close in distance and have high descriptor similarity, they are considered duplicates and one of them is retained to obtain the optimized feature point set.

[0067] In one embodiment, adjacent frame images corresponding to the target image are obtained, and adjacent frame feature point sets corresponding to the adjacent frame images are obtained; feature point distances between each optimized feature point in the optimized feature point set and each adjacent frame feature point in the adjacent frame feature point set are calculated, and based on the feature point distances, a set of matching feature point pairs between the optimized feature point set and the adjacent frame feature point set is determined; target matching feature points in the target image are extracted from the matching feature point pair set to obtain a feature point set corresponding to the target image.

[0068] Specifically, the same preprocessing operation as that of the target image is performed on the adjacent frame images, and the ORB algorithm is used to extract feature points in the reference image to obtain a set of adjacent frame feature points corresponding to the adjacent frame images.

[0069] Specifically, the Hamming distance calculation formula is used to calculate the feature point distance between each optimized feature point in the optimized feature point set and each adjacent frame feature point in the adjacent frame feature point set, and the feature point distance is compared with the preset feature point distance threshold. If the feature point distance is not greater than the preset feature point distance threshold, the optimized feature point and the adjacent frame feature point corresponding to the feature point distance are considered to be a matching feature point pair, and all matching feature point pairs are integrated to obtain a matching feature point pair set.

[0070] Specifically, the target matching feature points of the target image are extracted from the matching feature point pair set, and these target matching feature points constitute the feature point set of the target image after feature point extraction and feature point matching processing.

[0071] In one embodiment, multiple frames of associated images corresponding to the target image are obtained, and associated feature point sets corresponding to each of the multiple frames of associated images are obtained.

[0072] In one embodiment, feature point matching processing is performed on the feature point set corresponding to the target image and the associated feature point sets corresponding to each of the multiple frames of associated images to obtain a plurality of associated feature point matching point pairs, and based on the plurality of associated feature point matching point pairs, the posture transformation of the shooting device is determined, and based on the posture transformation and the plurality of associated feature point matching point pairs, an initial global map is constructed.

[0073] Specifically, the feature points of the target image are matched with the feature points of the multi-frame associated images respectively through the feature point matching algorithm; for the feature points in the target image, the associated feature points that match them are found in the feature point set of the multi-frame associated images to obtain multiple associated feature point matching point pairs.

[0074] Specifically, based on the matched point pairs of multiple associated feature points, the posture change of the shooting device between different frames is calculated, where the posture change includes translation and rotation information; the above process is repeated to match and calculate the posture of the feature points of multiple frames of images. By continuously accumulating this relative transformation information, the maps and feature points at different time points are aligned to achieve the fusion of multi-frame image information and determine the posture change of the shooting device.

[0075] Specifically, based on the posture transformation of the shooting device obtained by fusing multi-frame image information and multiple associated feature point matching pairs, the semantic SLAM system will gradually build a global map as the initial global map. The initial global map contains the position information of each feature point in the global coordinate system, as well as the posture information of the camera at different times.

[0076] In one embodiment, based on the target semantic information, the two-dimensional coordinates of the target are determined, and based on the image depth map, the depth information of the target is determined, based on the two-dimensional coordinates and the depth information, the target position of the target is determined, and based on the target position, the target is marked on the initial global map to update the initial global map.

[0077] Specifically, for each detected target, the target position in space is determined according to its position in the image and the depth information provided by the image depth map, and the target position combining semantic information and depth information is marked in the initial global map based on the existing feature points and spatial framework in the initial global map; preferably, the sparse point cloud generated by the semantic SLAM system can be combined to further refine the target position in the initial global map, so that the target position is integrated with the feature points in the map, the constructed environmental structure, etc.

[0078] In one embodiment, based on the target semantic information, a target category label is set for the target position corresponding to the target on the updated initial global map to generate a semantic label map.

[0079] Specifically, in the updated initial global map, the target location of each target is assigned a corresponding target category label. If a "table" appears at a specific location, then the location will be marked as M(x, y) = {c, if (x, y) is covered by the target; 0, otherwise}, where c is the target category label of the target.

[0080] Specifically, by semantically annotating all target locations in the initial global map, a semantic label map is generated. This semantic label map not only contains the spatial structure information of the environment, but also contains rich semantic information, which is helpful for subsequent tasks such as path planning and environmental understanding.

[0081] Step 103: extract and filter feature points of the target image to obtain a plurality of first feature points, calculate semantic weight values corresponding to each of the plurality of first feature points, and determine a moving path based on the semantic weight values and a preset path planning algorithm.

[0082] In one embodiment, conventional image processing methods, such as ORB, SIFT, etc., are used to extract feature points from an image; duplicate or blurred feature points are filtered out, and the most distinctive feature points are retained; when performing feature point screening, feature points located in semantically important areas are selected through target detection and semantic segmentation to form multiple first feature point sets, wherein the semantically important area is centered on the target detected by the target, and the area within its preset range is used as the semantically important area.

[0083] In one embodiment, the target categories corresponding to the multiple first feature points are respectively obtained, and based on the target categories, the target category weight values corresponding to the multiple first feature points are determined, and the distances between the multiple first feature points and the target to be photographed are respectively calculated, and based on the target category weight values and the distances, the semantic weight values corresponding to the multiple first feature points are determined.

[0084] Specifically, corresponding semantic weight values are set for all targets that may appear in the input image, wherein the semantic weight values are set according to the categories corresponding to the targets and their importance, such as the weight value of important target objects is greater than the weight value of non-target objects.

[0085] Specifically, based on the target categories corresponding to the multiple first feature points, a semantic weight value of the target corresponding to the target category is obtained, and the semantic weight value is set as the target category weight value corresponding to the multiple first feature points.

[0086] Specifically, based on a distance calculation formula, such as the Euclidean distance formula, the distances between the multiple first feature points and the target to be photographed are calculated respectively, wherein the target to be photographed is selected from all targets detected in the early stage of target detection.

[0087] Preferably, the path starting point and path end point of the path to be generated can also be obtained, and the path starting point distance and path end point distance corresponding to the multiple first feature points and the path starting point and the path end point can be calculated respectively. Based on the distance and the average value of the distance between the path starting point distance and the path end point distance, the distance corresponding to each of the multiple first feature points can be determined.

[0088] Specifically, the target category weight value and the distance are input into a preset semantic weight formula to calculate the semantic weight values corresponding to the plurality of first feature points, wherein the semantic weight formula is as follows:

[0089] w i =α·D(p i ,target)+β·C(object i );

[0090] Where w i is the semantic weight value corresponding to the first feature point i, α and β are adjustment coefficients, D(p i ,target) is the first feature point p i The distance to the target to be photographed, C(object i ) is the target category weight value corresponding to the i-th first eigenvalue.

[0091] In one embodiment, the path starting point and path end point of the path to be generated are obtained, and the target first feature point between the path starting point and the path end point is searched based on a preset path planning algorithm. An initial moving path is generated based on the path starting point, the path end point and the target first feature point, and the total cost of the first path corresponding to the initial moving path is calculated based on a path cost function.

[0092] Specifically, the preset path planning algorithms include but are not limited to the A* algorithm and the Dijkstra algorithm; wherein the A* algorithm guides the search by evaluating the cost function f(n) = g(n) + h(n), where g(n) is the actual cost from the starting point of the path to the node n, usually the Euclidean distance or the Manhattan distance, and h(n) is the heuristic estimated cost from the node n to the end point of the path, usually the straight-line distance.

[0093] Specifically, take algorithm A as an example to perform path search: Initialization: add the starting point of the path to the open list (used to store the nodes to be explored), and set the starting point g(starting point) = 0, h(starting point) as the straight-line distance from the starting point to the target point, f(starting point) = g(starting point) + h(starting point); Circular search: select the node n with the smallest f(n) value from the open list for exploration, check whether node n is the end point of the path, if so, the path search ends, and the preliminary path from the starting point to the end point of the path is obtained based on the recorded node path backtracking; otherwise, obtain the adjacent nodes m of node n, such as the adjacent nodes of node n in different directions; for each adjacent node m, calculate h(m) is the straight-line distance from the adjacent node m to the target point, f(m) = g(m) + h(m); if the adjacent node m is not in the open list, add it to the open list and record its predecessor node as node n; if the adjacent node m is already in the open list and the newly calculated g(m) is less than the original g(m) value, then update g(m), h(m), f(m) and the predecessor node to n.

[0094] Preferably, the node n mentioned in the A algorithm is selected from the multiple first feature points mentioned above.

[0095] Specifically, the path cost function is as follows:

[0096]

[0097] In the formula, C(P) is the total path cost, λ is the semantic weight coefficient, and w i is the semantic weight value corresponding to the i-th feature point constituting the path, (x i ,y i ) is the coordinate of the i-th feature point constituting the path.

[0098] Specifically, according to the semantic weight value w i and semantic weight coefficient w i , and adjust the path. If the path passes through an area with a high semantic weight, the total path cost will increase accordingly. In this case, you can search for a path with a lower total path cost. After the above path search and adjustment process, the path with a lower total path cost and meeting the task requirements is the final movement path.

[0099] In one embodiment, the path cost function is adjusted based on the target semantic information, and the initial moving path is optimized based on the adjusted path cost function to obtain an optimized moving path.

[0100] Specifically, when optimizing the initial moving path, after the preliminary path is generated, a topological graph can be constructed to represent the nodes and their connection relationships in the path space. The path can be further optimized through graph theory methods, such as the shortest path algorithm, and the relationship between edges and nodes in the topological graph can be used to smooth and adjust the preliminary path to avoid obstacles or inaccessible areas.

[0101] Specifically, semantic maps can be analyzed to incorporate semantic information as constraints during path planning. This includes semantic priority path selection, which prioritizes areas around target objects based on their semantic importance as nodes along the path; and dynamic obstacle consideration, which takes into account moving targets (such as humans and robots) or dynamic environments in real time and dynamically adjusts the path to avoid these obstacles.

[0102] Specifically, after A* or Dijkstra generates a preliminary path, the target semantic information is used to adjust the path cost function to increase or decrease the cost of certain paths. After the path cost function is adjusted, it can be optimized again to make the generated path more consistent with the semantic target, ensuring that the path is both smooth and meets actual needs. In the process of path optimization, if the path is too tortuous or irregular, a path smoothing algorithm (such as Bézier curves, Catmull-Romsplines, etc.) can be used to smooth the path to ensure smoother movement of the camera.

[0103] Specifically, the adjusted path cost function is as follows:

[0104]

[0105] Where C final (P) is the total cost of the optimized mobile path, λ is the semantic weight coefficient, w i is the semantic weight value corresponding to the i-th feature point constituting the path, (x i ,y i ) is the coordinate of the i-th feature point constituting the path, PathSmoothness is the measure of path smoothness, and α is the weighting coefficient of smoothness.

[0106] Specifically, the final path not only takes into account the physical layout of the environment, but also fully incorporates semantic information to ensure that the generated path not only meets the movement requirements of the device, but also can accurately guide to the target area, achieving the purpose of efficient shooting and navigation.

[0107] Step 104: Control the shooting device to move based on the moving path, and during the movement, obtain the environmental change detection amount in real time, dynamically adjust the moving path based on the environmental change detection amount to obtain a dynamically adjusted moving path, and control the shooting device to move based on the dynamically adjusted moving path.

[0108] In one embodiment, during the movement of the mobile path, by detecting changes in the environment in real time, such as the addition of new obstacles or the movement of targets, the path planning results are adjusted through dynamic updates of the local map and key frame optimization to ensure that the device can flexibly adapt to the changing environment.

[0109] In one embodiment, when obtaining the environmental change detection amount in real time, the current target semantic information of the first feature point in the current image frame and the previous target semantic information of the first feature point in the previous frame are obtained in real time, and the current target semantic information and the previous target semantic information are input into a preset environmental change detection formula to obtain the environmental change detection amount, wherein the preset environmental change detection formula is as follows:

[0110]

[0111] Where ΔM is the amount of environmental change detected, M t (p i ) is the current target semantic information, M t-1 (p i ) is the semantic information of the previous target, and N is the number of detected feature points.

[0112] Specifically, the environmental change detection amount is compared with a preset environmental change detection amount threshold. If the environmental change detection amount is not greater than the environmental change detection amount threshold, it means that the environmental change is small and there is no need to make major adjustments to the moving path; if the environmental change detection amount is greater than the environmental change detection amount threshold, it means that the environmental change is large and there is a need to make major adjustments to the moving path.

[0113] Specifically, the current position of the shooting device is used as a new path starting point, and the path end point remains unchanged. Based on the above method, the moving path is regenerated to obtain a dynamically adjusted moving path.

[0114] In one embodiment, after obtaining the moving path, a control instruction including linear velocity and angular velocity is generated based on the moving path, and the camera is controlled to move based on the moving path based on the control instruction so that the camera moves at an appropriate speed and direction.

[0115] Specifically, the system calculates the depth error and direction error based on the current position of the camera and the path points in the moving path, and generates control instructions for the linear velocity v and angular velocity ω.

[0116] Specifically, the linear velocity v and angular velocity ω are as follows:

[0117] v=k v ×(D t -D d );ω=k ω×(θ t -θ d );

[0118] Where v is the linear velocity, ω is the angular velocity, and D t is the current depth value, D d is the expected depth value, θ t is the current direction angle, θ d is the desired direction angle, k v and k ω These are the gain coefficients for linear velocity and angular velocity, respectively, used to adjust the device's response to errors.

[0119] Specifically, the screen of the shooting device displays the direction of the path in real time, such as superimposed arrows and target points. The system superimposes navigation arrows and path lines on the display screen and updates the guidance in real time. Users can adjust the movement of the shooting device through the guidance information on the screen to ensure the smoothness and accuracy of the shooting process.

[0120] In one embodiment, when it is detected that the shooting device has a shooting device deviation, the current field of view coverage of the shooting device is calculated based on a preset field of view coverage evaluation formula; when the current field of view coverage is less than a preset field of view coverage threshold, the offset angle between the target area center of the target and the field of view center of the shooting device is calculated, the offset angle is adjusted based on the offset angle, and an angle adjustment instruction is generated based on the adjusted offset angle, so that the shooting device is controlled to adjust the angle based on the angle adjustment instruction.

[0121] Specifically, when the device's posture deviates due to external disturbances or user operations, the system analyzes the distribution of feature points and the number of target areas, evaluates the coverage of the current field of view, and generates adjustment instructions to optimize the field of view coverage.

[0122] Specifically, the preset visual field coverage evaluation formula is as follows:

[0123]

[0124] Where, S(p i ) is the feature point p i The coverage area is calculated by the pixel count or bounding box area of the target area; S total is the total coverage area, which is determined by the field of view of the camera device, and V is the current field of view coverage.

[0125] Specifically, when calculating the adjusted offset angle based on the offset angle, the offset angle is input into a preset adjusted offset angle calculation formula to obtain the adjusted offset angle, wherein the adjusted offset angle calculation formula is as follows:

[0126] ω adjust =kω ×Δθ;

[0127] Where, ω adjust To adjust the offset angle, Δθ is the offset angle, k ω is the angular velocity gain coefficient.

[0128] In one embodiment, in order to improve the quality of subsequent three-dimensional modeling, the system filters and optimizes the images captured by the shooting device based on the moving path; by filtering out blurred or incomplete images, high-quality frames are stored and compressed to reduce storage space usage and improve the efficiency of subsequent data utilization.

[0129] Specifically, the blur detection score of each collected image is calculated using the blur detection formula. Based on the blur detection score, it is determined whether to save the current collected image. If the blur detection score is less than the blur detection score threshold, the current collected image is saved; otherwise, the current collected image is filtered.

[0130] Specifically, the fuzzy detection formula is as follows:

[0131]

[0132] Where B is the blur detection score, I(x,y) is the image grayscale value or pixel intensity value, which represents the brightness value of the image at position (x,y). is the gradient of the image in the horizontal direction (x-axis), that is, the rate of change of the image grayscale value in the horizontal direction, It is the gradient of the image in the vertical direction (y-axis), that is, the rate of change of the image grayscale value in the vertical direction.

[0133] Specifically, all saved collected images are compressed and stored using the JPEG2000 format.

[0134] In one embodiment, to ensure the shooting quality and stability of the device operation, the system also provides multiple real-time reminder functions, including but not limited to movement speed reminder, blur detection reminder and tilt angle reminder.

[0135] Specifically, regarding the movement speed reminder: if the device is detected to be moving too fast based on the movement speed formula, the system will remind the user to slow down. The movement speed formula is as follows:

[0136] V=Δd / Δt, if V>V max , prompting the user to slow down.

[0137] Specifically, for blur detection reminders: if the image quality is determined to be poor based on the blur detection formula, the user is prompted to hold the device steady.

[0138] Specifically, for the tilt angle reminder: if the tilt angle of the device is detected to exceed the preset threshold based on the tilt angle reminder formula, a prompt to adjust the direction is provided; wherein the tilt angle reminder formula is: Δθ=atan(a y / a x ); When Δθ exceeds the set threshold, the user is prompted to adjust the device through the interface.

[0139] In one embodiment, an intelligent shooting guidance system based on the combination of deep learning and semantic SLAM can efficiently realize environmental perception, path planning and three-dimensional modeling, and has a wide range of application scenarios, covering the following major areas: digital cultural relics protection, architectural and engineering surveying and mapping, urban planning and digital twin technology, emergency rescue and disaster assessment, and credit industry applications.

[0140] Example 2, see Figure 2 , Figure 2 is a schematic diagram of the structure of an embodiment of an intelligent photography guidance device provided by this application. Corresponding to the above intelligent photography guidance method, this application also provides an intelligent photography guidance device. This intelligent photography guidance device includes a unit for executing the above intelligent photography guidance method. This intelligent photography guidance device can be configured in a desktop computer, tablet computer, laptop computer, or other terminal. Specifically, this intelligent photography guidance device includes a semantic segmentation module 201, a semantic label map generation module 202, a movement path generation module 203, and a photography device movement control module 204.

[0141] The semantic segmentation module 201 is used to perform target detection and semantic segmentation processing on the input image based on the deep learning model to obtain a target image containing target semantic information.

[0142] The semantic label map generation module 202 is used to generate an image depth map corresponding to the target image based on a depth estimation model, perform feature point extraction and feature point matching on the target image to obtain a feature point set corresponding to the target image, generate an initial global map based on the feature point set, and generate a semantic label map based on the target semantic information, the image depth map and the initial global map.

[0143] The moving path generation module 203 is used to extract and screen feature points of the target image to obtain multiple first feature points, calculate the semantic weight values corresponding to each of the multiple first feature points, and determine the moving path based on the semantic weight values and a preset path planning algorithm.

[0144] The shooting device movement control module 204 is used to control the shooting device to move based on the movement path, and during the movement, obtain the environmental change detection amount in real time, dynamically adjust the movement path based on the environmental change detection amount, obtain a dynamically adjusted movement path, and control the shooting device to move based on the dynamically adjusted movement path.

[0145] In one embodiment, the semantic segmentation module 201 is used to perform target detection and semantic segmentation processing on the input image based on the deep learning model to obtain a target image containing target semantic information, specifically including: the deep learning model includes a target detection model and a semantic segmentation model; real-time acquisition of an image, preprocessing the image to obtain a first preprocessed image, inputting the first preprocessed image into the target detection model so that the target detection model performs feature extraction on the first preprocessed image to obtain an image feature map, performing grid division processing on the image feature map to obtain multiple feature grids, and performing target detection processing on the multiple feature grids respectively to obtain target corresponding The target category, target bounding box and target mask are obtained, and the pre-processed image is fused based on the target category, the target bounding box and the target mask to obtain a fused image; the fused image is input into the semantic segmentation model so that the semantic segmentation model performs pixel-level classification processing on the fused image to obtain the category label corresponding to each pixel point, generate a semantic segmentation category label map, and perform semantic information extraction on the semantic segmentation category label map to extract the target category and target boundary information corresponding to each target, and generate a target image containing target semantic information; wherein the target semantic information includes the target category and the target boundary information corresponding to each target.

[0146] In one embodiment, the semantic label map generation module 202 is used to perform feature point extraction and feature point matching processing on the target image to obtain a feature point set corresponding to the target image, specifically including: performing corner point detection on the target image based on the ORB algorithm to obtain multiple corner points in the target image, and respectively obtaining the corner point directions and corner point descriptors corresponding to each of the multiple corner points; integrating the corner point directions and corner point descriptors corresponding to each of the multiple corner points to generate an initial feature point set; based on the target semantic information in the target image, screening the initial feature point set to obtain an optimized feature point set; obtaining adjacent frame images corresponding to the target image, and obtaining an adjacent frame feature point set corresponding to the adjacent frame images; calculating the feature point distance between each optimized feature point in the optimized feature point set and each adjacent frame feature point in the adjacent frame feature point set, and determining a matching feature point pair set between the optimized feature point set and the adjacent frame feature point set based on the feature point distance; extracting the target matching feature points in the target image from the matching feature point pair set to obtain a feature point set corresponding to the target image.

[0147] In one embodiment, the semantic label map generation module 202 is used to generate an initial global map based on the feature point set, and to generate a semantic label map based on the target semantic information, the image depth map and the initial global map, specifically including: obtaining multiple frames of associated images corresponding to the target image, and obtaining the associated feature point sets corresponding to each of the multiple frames of associated images; performing feature point matching processing on the feature point set corresponding to the target image and the associated feature point set corresponding to each of the multiple frames of associated images, respectively, to obtain multiple associated feature point matching point pairs, and determining the shooting location based on the multiple associated feature point matching point pairs. The method comprises the following steps: determining the posture transformation of the device, and constructing an initial global map based on the posture transformation and the multiple associated feature point matching pairs; determining the two-dimensional coordinates of the target based on the target semantic information, and determining the depth information of the target based on the image depth map, determining the target position of the target based on the two-dimensional coordinates and the depth information, and marking the target on the initial global map based on the target position to update the initial global map; setting a target category label for the target position corresponding to the target on the updated initial global map based on the target semantic information to generate a semantic label map.

[0148] In one embodiment, the mobile path generation module 203 is used to calculate the semantic weight value corresponding to each of the multiple first feature points, and determine the mobile path based on the semantic weight value and a preset path planning algorithm, specifically including: respectively obtaining the target category corresponding to each of the multiple first feature points, and determining the target category weight value corresponding to the multiple first feature points based on the target category, and respectively calculating the distance between the multiple first feature points and their corresponding targets, and determining the semantic weight value corresponding to the multiple first feature points based on the target category weight value and the distance; obtaining the path starting point and path end point of the path to be generated, searching for the target first feature point between the path starting point and the path end point based on a preset path planning algorithm, and generating an initial mobile path based on the path starting point, the path end point and the target first feature point, and calculating the first path total cost corresponding to the initial mobile path based on a path cost function; adjusting the path cost function based on the target semantic information, and optimizing the initial mobile path based on the adjusted path cost function to obtain an optimized mobile path.

[0149] In one embodiment, the semantic label map generation module 202 is used to generate an image depth map corresponding to the target image based on a depth estimation model, specifically including: performing image preprocessing on the target image to obtain a second preprocessed image; inputting the second preprocessed image into the depth estimation model so that the depth estimation model predicts a first distance from each pixel in the second preprocessed image to the shooting device, and generating a first image depth map based on the first distance; obtaining multiple frames of continuous images corresponding to the target image, and obtaining the first continuous image depth map corresponding to the multiple frames of continuous images; fusing the first continuous image depth map and the first image depth map to obtain a first fused depth map, and filtering the first fused depth map to obtain an image depth map corresponding to the target image.

[0150] In one embodiment, the intelligent guided shooting device provided by the present application further includes: a shooting equipment adjustment module.

[0151] In one embodiment, the shooting device adjustment module is used to calculate the current field of view coverage of the shooting device based on a preset field of view coverage evaluation formula when it is detected that the shooting device has a shooting device deviation; when the current field of view coverage is less than a preset field of view coverage threshold, calculate the offset angle between the target area center of the target and the field of view center of the shooting device, calculate the adjusted offset angle based on the offset angle, and generate an angle adjustment instruction based on the adjusted offset angle, so as to control the shooting device to adjust the angle based on the angle adjustment instruction.

[0152] The above-mentioned intelligent shooting guidance device can implement the intelligent shooting guidance method of the above-mentioned method embodiment. The optional options in the above-mentioned method embodiment are also applicable to this embodiment and will not be described in detail here.

[0153] like Figure 3 As shown, Figure 3 This is a structural diagram of an electronic device provided by the present application; it includes a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114, and the memory 113 is used to store computer programs.

[0154] In one embodiment of the present application, the processor 111 is configured to implement the intelligent shooting guidance method provided by any one of the aforementioned method embodiments when executing a program stored in the memory 113 .

[0155] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0156] Therefore, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the intelligent shooting guidance method provided in any of the aforementioned method embodiments are implemented.

[0157] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, etc. Any physical storage medium capable of storing program code can be non-volatile or volatile.

[0158] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0159] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and other division methods may be used in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not implemented.

[0160] The steps in the method of the embodiment of the present application can be adjusted in order, combined, and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided, and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit.

[0161] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0162] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0163] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, as long as these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

[0164] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An intelligent shooting guidance method, characterized in that: include: Perform target detection and semantic segmentation on the input image based on the deep learning model to obtain a target image containing target semantic information; generating an image depth map corresponding to the target image based on a depth estimation model, performing feature point extraction and feature point matching on the target image to obtain a feature point set corresponding to the target image, generating an initial global map based on the feature point set, and generating a semantic label map based on the target semantic information, the image depth map, and the initial global map; Performing feature point extraction and feature point screening processing on the target image to obtain a plurality of first feature points, calculating semantic weight values corresponding to each of the plurality of first feature points, and determining a movement path based on the semantic weight values and a preset path planning algorithm; The shooting device is controlled to move based on the moving path, and during the movement, the environmental change detection amount is obtained in real time. Based on the environmental change detection amount, the moving path is dynamically adjusted to obtain a dynamically adjusted moving path, and the shooting device is controlled to move based on the dynamically adjusted moving path.

2. The method according to claim 1, wherein: Based on the deep learning model, the input image is processed for target detection and semantic segmentation to obtain a target image containing target semantic information, specifically including: The deep learning model includes a target detection model and a semantic segmentation model; Acquiring an image in real time, preprocessing the image to obtain a first preprocessed image, inputting the first preprocessed image into the target detection model so that the target detection model performs feature extraction on the first preprocessed image to obtain an image feature map, performing grid division processing on the image feature map to obtain multiple feature grids, performing target detection processing on each of the multiple feature grids to obtain a target category, a target bounding box, and a target mask corresponding to the target, and fusing the preprocessed image based on the target category, the target bounding box, and the target mask to obtain a fused image; The fused image is input into the semantic segmentation model so that the semantic segmentation model performs pixel-level classification processing on the fused image to obtain a category label corresponding to each pixel point, generate a semantic segmentation category label map, and perform semantic information extraction on the semantic segmentation category label map to extract the target category and target boundary information corresponding to each target, and generate a target image containing target semantic information; wherein the target semantic information includes the target category and the target boundary information corresponding to each target.

3. The method according to claim 1, wherein Performing feature point extraction and feature point matching on the target image to obtain a feature point set corresponding to the target image, specifically including: Performing corner detection on the target image based on the ORB algorithm to obtain multiple corner points in the target image, and respectively obtaining corner point directions and corner point descriptors corresponding to each of the multiple corner points; Integrating the corner point directions and corner point descriptors corresponding to each of the multiple corner points to generate an initial feature point set; Based on the target semantic information in the target image, the initial feature point set is screened to obtain an optimized feature point set; Acquire adjacent frame images corresponding to the target image, and acquire adjacent frame feature point sets corresponding to the adjacent frame images; Calculating the feature point distances between each optimized feature point in the optimized feature point set and each adjacent frame feature point in the adjacent frame feature point set, and determining a set of matching feature point pairs between the optimized feature point set and the adjacent frame feature point set based on the feature point distances; Target matching feature points in the target image are extracted from the set of matching feature point pairs to obtain a feature point set corresponding to the target image.

4. The method according to claim 3, wherein: Generating an initial global map based on the feature point set, and generating a semantic label map based on the target semantic information, the image depth map, and the initial global map, specifically comprising: Acquire multiple frames of associated images corresponding to the target image, and acquire associated feature point sets corresponding to each of the multiple frames of associated images; Performing feature point matching processing on the feature point set corresponding to the target image and the associated feature point sets corresponding to each of the multiple frames of associated images to obtain a plurality of associated feature point matching pairs, determining a posture transformation of the camera based on the plurality of associated feature point matching pairs, and constructing an initial global map based on the posture transformation and the plurality of associated feature point matching pairs; Determining the two-dimensional coordinates of the target based on the target semantic information, and determining the depth information of the target based on the image depth map, determining the target position of the target based on the two-dimensional coordinates and the depth information, and marking the target on the initial global map based on the target position to update the initial global map; Based on the target semantic information, a target category label is set for the target position corresponding to the target on the updated initial global map to generate a semantic label map.

5. The method according to claim 1, wherein: Calculating semantic weight values corresponding to each of the plurality of first feature points, and determining a movement path based on the semantic weight values and a preset path planning algorithm, specifically comprising: Obtaining target categories corresponding to the plurality of first feature points respectively, and determining target category weight values corresponding to the plurality of first feature points based on the target categories, and calculating distances between the plurality of first feature points and the target to be photographed respectively, and determining semantic weight values corresponding to the plurality of first feature points based on the target category weight values and the distances; Obtaining a path starting point and a path end point of a to-be-generated path, searching for a target first feature point between the path starting point and the path end point based on a preset path planning algorithm, generating an initial movement path based on the path starting point, the path end point, and the target first feature point, and calculating a first path total cost corresponding to the initial movement path based on a path cost function; The path cost function is adjusted based on the target semantic information, and the initial moving path is optimized based on the adjusted path cost function to obtain an optimized moving path.

6. The method according to claim 1, wherein: Generating an image depth map corresponding to the target image based on the depth estimation model specifically includes: Performing image preprocessing on the target image to obtain a second preprocessed image; Inputting the second pre-processed image into a depth estimation model so that the depth estimation model predicts a first distance from each pixel in the second pre-processed image to the camera, and generating a first image depth map based on the first distance; Acquire multiple frames of continuous images corresponding to the target image, and acquire depth maps of the first continuous images corresponding to the multiple frames of continuous images; The first continuous image depth map and the first image depth map are fused to obtain a first fused depth map, and the first fused depth map is filtered to obtain an image depth map corresponding to the target image.

7. The method according to claim 1, wherein: Also includes: When it is detected that the shooting device is deviated, the current field of view coverage of the shooting device is calculated based on a preset field of view coverage evaluation formula; When the current field of view coverage is less than a preset field of view coverage threshold, the offset angle between the target area center of the target and the field of view center of the shooting device is calculated, the offset angle is adjusted based on the offset angle, and an angle adjustment instruction is generated based on the adjusted offset angle, so that the shooting device is controlled to adjust the angle based on the angle adjustment instruction.

8. An intelligent shooting guidance device, characterized in that: include: Semantic segmentation module, semantic label map generation module, movement path generation module and camera movement control module; The semantic segmentation module is used to perform target detection and semantic segmentation processing on the input image based on the deep learning model to obtain a target image containing target semantic information; The semantic label map generation module is used to generate an image depth map corresponding to the target image based on a depth estimation model, perform feature point extraction and feature point matching on the target image to obtain a feature point set corresponding to the target image, generate an initial global map based on the feature point set, and generate a semantic label map based on the target semantic information, the image depth map, and the initial global map; The movement path generation module is configured to perform feature point extraction and feature point screening processing on the target image to obtain a plurality of first feature points, calculate semantic weight values corresponding to each of the plurality of first feature points, and determine a movement path based on the semantic weight values and a preset path planning algorithm; The shooting device movement control module is used to control the shooting device to move based on the movement path, and during the movement, obtain the environmental change detection amount in real time, dynamically adjust the movement path based on the environmental change detection amount, obtain a dynamically adjusted movement path, and control the shooting device to move based on the dynamically adjusted movement path.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the computer program can implement the method according to any one of claims 1 to 7.

Images