Target positioning method and device based on image segmentation and medium
By combining the calibration relationship between two-dimensional images and three-dimensional point clouds, using improved neural networks and multimodal deep learning networks, the precise positioning of hidden danger targets in complex scenarios is achieved, and the recognition efficiency and accuracy of traditional algorithms in complex environments is solved. The problem of insufficient improvements in processing efficiency and accuracy.
Patent Information
- Application Number
- CN202410959295.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-15
AI Technical Summary
In complex environments, traditional image segmentation algorithms are difficult to achieve efficient identification of small targets such as power transmission lines and flames, and the processing speed is slow and it is difficult to achieve the expected effect, especially in complex scenarios, which are insufficient recognition accuracy and difficult to achieve profound scenes. understand.
Using the target positioning method based on image segmentation, data dimensionality reduction is performed by acquiring two-dimensional images and three-dimensional point clouds, using the improved first neural network and multimodal deep learning network, combining the calibration relationship between two-dimensional images and three-dimensional point clouds. Classify to achieve accurate positioning of hidden danger targets in complex scenarios.
The processing efficiency and recognition accuracy are improved, and the point cloud is initially screened through efficient image segmentation to achieve accurate positioning of hidden danger targets in complex scenarios, improving the system's processing efficiency and recognition accuracy.
Smart Images

Figure CN118968015A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and in particular to a target positioning method, device and medium based on image segmentation. Background Art
[0002] With the development of technology, it is possible to locate targets based on image segmentation. Common semantic segmentation models include U-Net, DeepLab, and YOLO series. Among them, the U-Net model adopts an encoder-decoder structure, and retains feature information of different resolutions through skip connections, thereby improving segmentation accuracy. DeepLab uses dilated convolution to expand the receptive field, thereby increasing the contextual information of each pixel and improving segmentation accuracy. The YOLO series of algorithms has good detection effects in vehicle target detection with its end-to-end advantages and fast detection speed.
[0003] However, in some complex environments, traditional algorithm models still face some challenges, including environmental adaptability, target recognition accuracy, target diversity, model lightweight, and deep understanding of the scene. There are still some shortcomings.
[0004] For example, with the continuous development of the power system, the frequency of various engineering operations near transmission lines has also increased significantly. It is very easy for engineering vehicles to touch the transmission lines due to improper operation, which may cause damage to the transmission lines and casualties, seriously affecting the normal operation of the power grid. Therefore, it is particularly important to locate and segment potential hazards in complex scenarios.
[0005] However, in the transmission line environment, the recognition environment is complex, and it is necessary to identify small targets such as lines and flames. Traditional algorithm models need to process a large amount of data, the processing speed is slow, and it is difficult to achieve the expected effect. Other scenes, such as construction sites and crowded scenes, also have similar problems. Summary of the invention
[0006] In order to solve the above problems, this application proposes a target positioning method based on image segmentation, including:
[0007] Collect the 2D image and 3D point cloud corresponding to the target scene;
[0008] Based on the trained first neural network, performing target recognition segmentation on the two-dimensional image to determine a first position area of the recognized target in the two-dimensional image;
[0009] Based on the calibration relationship between the two-dimensional image and the three-dimensional point cloud, projecting the three-dimensional point cloud into the two-dimensional image, and performing data dimensionality reduction on the three-dimensional point cloud based on the first position area;
[0010] Based on the trained multimodal deep learning network, the three-dimensional point cloud after data dimensionality reduction is classified, and the second position area of the identified target in the three-dimensional point cloud is output based on the classification result.
[0011] On the other hand, the present application also proposes a target positioning device based on image segmentation, comprising:
[0012] at least one processor; and,
[0013] a memory communicatively connected to the at least one processor; wherein,
[0014] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the target positioning method based on image segmentation as described in the above example.
[0015] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as: the target positioning method based on image segmentation described in the above example.
[0016] The target localization method based on image segmentation proposed in this application can bring the following beneficial effects:
[0017] First, a more efficient image segmentation method is used to preliminarily screen the point cloud, and then the small-range point cloud data within the segmented area is classified to achieve accurate positioning of hidden danger targets in complex scenes, greatly improving the processing efficiency of the entire system.
[0018] By utilizing a multimodal deep learning network, we not only consider the information of the point cloud itself, but also integrate multimodal information to achieve accurate classification of the point cloud and increase the positioning accuracy of the identified target. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0020] Figure 1 Schematic diagram of the process of the target positioning method based on image segmentation in the embodiment of the present application;
[0021] Figure 2 This is a schematic diagram of the structure of the first neural network in the embodiment of the present application;
[0022] Figure 3 Schematic diagram of a target positioning device based on image segmentation in an embodiment of the present application. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0024] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0025] like Figure 1 As shown, the embodiment of the present application provides a method for target positioning based on image segmentation, comprising:
[0026] S101: Collecting a two-dimensional image and a three-dimensional point cloud corresponding to a target scene.
[0027] The target scene can be a transmission line scene, or it can be other scenes with more complex recognition environments, such as construction site scenes, industrial warehouse environments, traffic road scenes, etc. In the embodiments of the present application, the transmission line scene is used as an example for explanation.
[0028] An image acquisition device is pre-set in the target scene, wherein the two-dimensional image can be acquired by a conventional camera, and the three-dimensional point cloud can be acquired based on a depth camera.
[0029] S102: Based on the trained first neural network, perform target recognition segmentation on the two-dimensional image to determine a first position area of the recognized target in the two-dimensional image.
[0030] Generally speaking, a conventional neural network (for example, U-Net, DeepLab, YOLO series, etc.) can be used to analyze a two-dimensional image to identify the target and determine its corresponding first position area.
[0031] For example, after acquiring the image data of the target scene, the data is preprocessed (including data labeling, data enhancement, etc.) to obtain training samples, a model is built based on the corresponding neural network architecture, and the model is trained with the training samples to obtain a first neural network for identifying and segmenting the target.
[0032] However, the above model has limited recognition ability for complex scenes and it is difficult to achieve the expected effect. Based on this, the first neural network is improved so that it has stronger recognition ability in two-dimensional images.
[0033] It should be noted that in the embodiment of the present application, the use of a conventional neural network can also improve the problem that the target recognition capability in the traditional solution is difficult to achieve the expected effect, and the use of the first neural network in the embodiment of the present application can further improve the expected recognition effect.
[0034] Specifically, the backbone and head of the first neural network are constructed based on the YOLO series algorithm, for example, it can be the YOLOv8x algorithm. The backbone and head include convolutional layers, pooling layers, etc.
[0035] like Figure 2 As shown, the first neural network includes: a feature extraction network (Feature Extraction Network, FEN) and a cross-level feature fusion network (Cross-level Feature Fusion Network, CFFN).
[0036] The feature extraction network contains multiple layers from top to bottom, and different layers correspond to different resolutions. Figure 2 As shown, there are four levels in total, and the corresponding resolutions from top to bottom are: 160*160, 80*80, 40*40, 20*20, respectively representing features of different resolutions.
[0037] In a cross-level feature fusion network, there are multiple stages, each of which has a level corresponding to at least part of the multiple levels. Figure 2 As shown (in Figure 2 In the figure, C represents tensor calculation), which includes two stages. In the first stage, there are multiple levels from bottom to top. Each level performs feature fusion based on the output features of the previous level in this stage and the output features of the same level of the feature extraction network, and outputs the feature fusion result. In this stage, there is no level corresponding to the resolution of 160*160.
[0038] In the second stage, there are multiple layers from top to bottom. Each layer performs feature fusion based on the output features of the previous layer in this stage, the output features of the previous layer in the first stage, and the output features of the same layer of the feature extraction network, and outputs the feature fusion result. Among them, for the layer corresponding to the resolution of 160*160, it does not have the output features of the previous layer in the first stage, so it only needs to be fused through the other two features.
[0039] Furthermore, C2f_faster modules are set in each level of the feature extraction network and the cross-level feature fusion network to improve the reasoning speed and feature fusion capability of the model, which can achieve accurate segmentation of similar foreground and background points of hidden danger targets and has good system robustness.
[0040] In each level of the cross-level feature fusion network, a coordinate attention module is set in the module that needs to be output, so as to enhance the image segmentation model's ability to extract hidden danger target features in pictures under complex scenes, improve the model's attention to target features, and enable the model to focus more on the most critical information for the current task.
[0041] In the first stage, each layer is equipped with an upsampling module. Upsampling refers to the process of increasing a low-resolution image to a high-resolution or high-density sampled image. Its function is to increase the details and clarity of the image, thereby improving the image quality.
[0042] Furthermore, the coordinate attention module generates grid coordinate information for the input features, and performs mean pooling operations along the horizontal and vertical directions for the grid coordinate information to obtain corresponding one-dimensional mean pooling results.
[0043] By introducing grid coordinate information, the attention mechanism is extended from the pixel level to the image coordinate level. It can capture the global spatial structure of the image, pay more attention to the location information of the image, and enhance the understanding and representation ability of the image semantic information.
[0044] For the feature vector corresponding to the input feature, each channel is encoded along the horizontal coordinate (x, 1) and the vertical coordinate (1, y). This encoding method captures the long-range dependencies of spatial information in each channel by introducing a one-dimensional mean pooling operation in the horizontal and vertical directions.
[0045] Taking the horizontal direction as an example, firstly, the data in the column (0, 1) is averaged and pooled, and then the data in the column (1, 1) is averaged and pooled, and so on, so as to obtain the one-dimensional average pooling in the horizontal direction, and then the one-dimensional average pooling in the vertical direction is obtained in a similar way.
[0046] The one-dimensional mean pooling results corresponding to the horizontal and vertical directions are spliced, and coordinate attention is generated based on the splicing results, and the number of channels is reduced according to the two-dimensional convolution.
[0047] In the coordinate attention generation stage, the encoded features are concatenated along the spatial dimension, that is, the one-dimensional mean pooling results are concatenated. The number of channels is reduced through two-dimensional convolution to reduce the complexity of the model. The features are regularized and activated by applying batch normalization and nonlinear activation functions to generate the weights of coordinate attention.
[0048] The output of coordinate attention can be expressed by the following formula. The attention mechanism is applied in both horizontal and vertical directions, and the elements of each attention map reflect whether the object of interest exists in the corresponding row and column.
[0049] The formula is:
[0050]
[0051] Among them, Y v (i, j) refers to the output features, δ refers to the regularization and nonlinear activation operations, σ refers to the activation function operation, Conv refers to the convolution operation, Refers to the horizontal characteristics. Refers to the characteristics in the vertical direction, m is the length in the horizontal direction, and n is the length in the vertical direction.
[0052] In the coordinate attention module, through two one-dimensional global pooling operations, the network can obtain a larger receptive field and accurately encode spatial position information. It not only pays attention to the importance of the relationship between channels, but also takes into account the effective encoding of spatial information.
[0053] The coordinate attention mechanism models the correlation between features in the channel dimension, while capturing the relationship between different positions in the spatial dimension, improving the understanding and modeling capabilities of image features. When processing large-size images, the coordinate attention mechanism has good scalability and can effectively handle complex scenes and image situations.
[0054] In addition, in the lowest level of the feature extraction network, the output features pass through the fast spatial pyramid pooling SPPF module and are then input into the lowest level of the first stage of the cross-level feature fusion network.
[0055] In addition, a small target segmentation head module (HEAD) is provided, which includes multiple levels, each of which is connected to the corresponding level of the second stage, for example Figure 2 As shown in the figure, there are four levels, corresponding to the levels of the second stage. It provides richer feature information for the segmentation of small targets. The segmentation head is connected to the cross-level feature fusion network structure, and the model's segmentation and detection capabilities for objects of different scales are enhanced by using shallow information and additional feature fusion networks. In particular, the newly added scale segmentation head not only uses the shallow information of the input image, but also effectively transfers the feature information of small targets to the feature segmentation layers of the other three levels through the improved feature fusion layer, thereby significantly improving the positioning and segmentation accuracy of small targets.
[0056] S103: Based on the calibration relationship between the two-dimensional image and the three-dimensional point cloud, project the three-dimensional point cloud into the two-dimensional image, and perform data dimensionality reduction on the three-dimensional point cloud based on the first position area.
[0057] The calibration relationship between the 2D image and the 3D point cloud is usually generated by jointly calibrating the two, and is mainly used to represent the positional relationship between each point in the 2D image and each point in the 3D point cloud. Data dimensionality reduction refers to the screening of the number of points in the 3D point cloud.
[0058] Specifically, the corresponding feature points between the two-dimensional image and the three-dimensional point cloud are determined by the calibration plate. Feature points may include edges, corners, and points corresponding to certain targets of the image. The feature points of the calibration plate may be found by computer vision techniques (e.g., feature point detection algorithms), and the corresponding 3D points may be found in the three-dimensional point cloud by matching these feature points.
[0059] Based on the coordinates corresponding to the feature points, the internal and external parameters of the image acquisition device are calculated. The internal parameters may include the focal length of the camera, the optical center (the position of the center point of the image), and other parameters, which describe the geometric characteristics of the camera. The external parameters describe the position and direction of the camera in the world coordinate system, that is, the position and orientation of the camera, which may include rotation and translation parameters.
[0060] Through the known geometric information provided by the calibration board, combined with the feature points in the two-dimensional image and the corresponding 3D points in the three-dimensional point cloud, the internal and external parameters of the camera are calculated, so that the calibration relationship between the two-dimensional image and the three-dimensional point cloud is established through the internal and external parameters, and the geometric transformation relationship between the two-dimensional image and the three-dimensional point cloud is established.
[0061] After the calibration is completed and the calibration relationship is obtained, the calibration relationship can be used to project the 3D point cloud onto the 2D image, or to back-project the 2D image points into the 3D point cloud.
[0062] The three-dimensional point cloud is preprocessed, wherein the preprocessing includes at least one of normalization processing and filtering processing. At this time, a projection plane is determined, and the projection plane is the shooting plane of the two-dimensional image relative to the three-dimensional point cloud. According to the projection plane, the three-dimensional point cloud is mapped in the two-dimensional image.
[0063] Since the first neural network has been used to screen out the first position area of the identified target in the two-dimensional image, the specified point cloud belonging to the first position area can be screened out in the three-dimensional point cloud, and the remaining point clouds can be deleted to complete the first round of data dimensionality reduction processing.
[0064] According to the calibration relationship, the specified point cloud is reversely mapped back to the three-dimensional space, and then based on the point cloud optimization algorithm, the specified point cloud is optimized for data dimensionality reduction, and a second round of data dimensionality reduction processing is performed. In this way, a three-dimensional point cloud of tens of millions can be reduced to a three-dimensional point cloud of one hundred thousand, thereby significantly improving the processing efficiency and real-time performance of the system, which is suitable for real-time application scenarios.
[0065] Among them, the point cloud optimization algorithm is used to process the three-dimensional point cloud, thereby reducing the amount of data while retaining the main features of the data and deleting redundant point clouds. It can include voxel grid downsampling technology, filtering algorithm, point cloud registration algorithm, etc.
[0066] S104: Based on the trained multimodal deep learning network, classify the three-dimensional point cloud after data dimensionality reduction, and output the second position area of the identified target in the three-dimensional point cloud based on the classification result.
[0067] Multimodal deep learning network refers to an algorithm model that can recognize and process multimodal information corresponding to three-dimensional point clouds and classify three-dimensional point clouds. Multimodality can represent the coordinate information, color information, etc. of each point in the three-dimensional point cloud.
[0068] Similar to the first neural network, the three-dimensional point cloud can also be analyzed through a traditional neural network model (for example, PointNet, Voxel-based, etc.) to classify the point cloud and output a second position area corresponding to the identified target.
[0069] However, the above model has limited recognition ability for complex scenes and it is difficult to achieve the expected effect. Based on this, the traditional neural network model is improved so that it has stronger recognition ability in three-dimensional point clouds.
[0070] It should also be noted that in the embodiments of the present application, the use of conventional neural networks can also improve the problem that the target recognition ability in traditional solutions is difficult to achieve the expected effect, and the use of the multimodal deep learning network in the embodiments of the present application can further improve the expected recognition effect.
[0071] Specifically, the training process of the multimodal deep learning network includes: obtaining training samples, the training samples include a three-dimensional point cloud corresponding to the target scene. For the three-dimensional point cloud, each point contained therein is assigned corresponding modal information, and the modal information includes: coordinate information, spectral information, and geographic information corresponding to the target scene.
[0072] A deep learning network is constructed based on the second neural network, and multiple network layers are constructed in the deep learning network so that each network layer processes different modal information. The second neural network can be a neural network with three-dimensional point cloud processing capabilities such as PointNet, a 3D convolutional neural network, or a graph convolutional neural network.
[0073] The calculation results of multiple network layers are feature fused. After each network layer processes its own corresponding modal information, the calculation results of each network layer can be fused through weighted fusion, attention mechanism, etc., so as to realize the architecture of multimodal deep learning network. Then, through training samples, the multimodal deep learning network is trained using self-supervised learning method.
[0074] Furthermore, the network architecture of the multimodal deep learning network includes: an input layer, a feature extraction layer, a graph convolution layer, a multimodal fusion layer, a deep feature extraction layer, a classification layer, and an output layer, and the layers are arranged in this order.
[0075] The input layer is used to receive the input 3D point cloud, and the feature extraction layer includes a 3D convolution layer for extracting the local geometric features of the input data. The graph convolution layer is used to extract the global topological features of the input data based on the local geometric features.
[0076] Among them, multiple network layers are set in the feature extraction layer and the graph convolution layer, and the multimodal fusion layer is used to perform feature fusion on the calculation results of multiple network layers. In the specific execution, the feature extraction layer extracts the local geometric features corresponding to each modal information based on the input data transmitted by the input layer through the multiple network layers it contains, and fuses them through the multimodal fusion layer to obtain the final first fusion result. Then the first fusion result is used as input and input into the graph convolution layer. The graph convolution layer also has multiple network layers. Based on the input data (that is, the first fusion result), the global topological features corresponding to each modal information are extracted, and then they are fused through the multimodal fusion layer to obtain the final second fusion result.
[0077] At this time, the second fusion result is used as the input of the deep feature extraction layer, which includes two fully connected layers for extracting deep features and transmitting the extracted deep features to the classification layer. The classification layer is used to output the classification results of each point, so that the output layer outputs the second position area corresponding to the final recognition target according to the classification results.
[0078] Of course, in some cases, it is also possible to only set the multimodal fusion layer after the graph convolution layer and delete the multimodal fusion after the feature extraction layer.
[0079] In one embodiment, the multimodal data received by the input layer includes: coordinate data of a three-dimensional point cloud (in the shape of N x 3), spectral data (in the shape of N x C, where C is determined according to the expression mode of the point cloud color, and in the RGB mode, C is 3), and geographic information (in the shape of N x G, where G depends on the dimension of the modal information of the selected geographic information, which may include: topography, geology, etc.).
[0080] In the feature extraction layer, the 3D convolution layer extracts local geometric features through convolution operations, and the output shape is N x 64. The graph convolution layer extracts global topological features through graph convolution operations, and the output shape is N x 128. Among them, the multimodal fusion layer fuses data features of different modalities (coordinate data, spectral data, geographic information) to form a unified feature representation, and the output shape is N x 256.
[0081] The deep feature extraction layer further extracts deep features through two fully connected layers, with output shapes of N x 512 and N x 256 respectively.
[0082] The self-supervised pre-training module uses self-supervised learning methods to perform pre-training on large-scale unlabeled datasets to optimize feature representation. The features are input into the classification layer activated by Softmax, and the classification results of each point are output in the shape of N xK, where K is the number of categories. The output layer finally outputs the classification probability distribution of each point to achieve accurate classification of the point cloud.
[0083] like Figure 3 As shown, the embodiment of the present application also proposes a target positioning device based on image segmentation, including:
[0084] at least one processor; and,
[0085] a memory communicatively connected to the at least one processor; wherein,
[0086] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target positioning method based on image segmentation as described in any of the above embodiments.
[0087] The embodiment of the present application further proposes a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as: the target positioning method based on image segmentation described in any of the above embodiments.
[0088] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0089] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.
[0090] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0091] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0094] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0095] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0096] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0097] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0098] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A target positioning method based on image segmentation, characterized in that: include: Collect the 2D image and 3D point cloud corresponding to the target scene; Based on the trained first neural network, performing target recognition segmentation on the two-dimensional image to determine a first position area of the recognized target in the two-dimensional image; Based on the calibration relationship between the two-dimensional image and the three-dimensional point cloud, projecting the three-dimensional point cloud into the two-dimensional image, and performing data dimensionality reduction on the three-dimensional point cloud based on the first position area; Based on the trained multimodal deep learning network, the three-dimensional point cloud after data dimensionality reduction is classified, and the second position area of the identified target in the three-dimensional point cloud is output based on the classification result.
2. The method according to claim 1, characterized in that Based on the calibration relationship between the two-dimensional image and the three-dimensional point cloud, the three-dimensional point cloud is projected into the two-dimensional image, and data dimension reduction is performed on the three-dimensional point cloud based on the first position area, specifically including: Determine corresponding feature points between the two-dimensional image and the three-dimensional point cloud by using a calibration plate; Based on the coordinates corresponding to the feature points, the intrinsic parameters and the extrinsic parameters of the image acquisition device are calculated, and a calibration relationship between the two-dimensional image and the three-dimensional point cloud is established through the intrinsic parameters and the extrinsic parameters; Preprocessing the three-dimensional point cloud; wherein the preprocessing includes at least one of normalization processing and filtering processing; Determine a projection plane, and map the three-dimensional point cloud into the two-dimensional image according to the projection plane; Screening out a designated point cloud belonging to the first position area in the three-dimensional point cloud, and reversely mapping the designated point cloud back to the three-dimensional space according to the calibration relationship; Based on the point cloud optimization algorithm, data dimension reduction optimization is performed on the specified point cloud.
3. The method according to claim 1, characterized in that The training process of the multimodal deep learning network includes: Acquire a training sample, wherein the training sample includes a three-dimensional point cloud corresponding to the target scene; For the three-dimensional point cloud, assign corresponding modal information to each point contained therein, wherein the modal information includes: coordinate information, spectral information, and geographic information corresponding to the target scene; Building a deep learning network based on the second neural network, and building multiple network levels in the deep learning network so that each network level processes different modal information; The calculation results of the multiple network levels are feature fused, and a multimodal deep learning network is trained using the training samples and a self-supervised learning method.
4. The method according to claim 3, characterized in that The network architecture of the multimodal deep learning network includes: an input layer, a feature extraction layer, a graph convolution layer, a multimodal fusion layer, a deep feature extraction layer, a classification layer, and an output layer; The feature extraction layer includes a 3D convolution layer for extracting local geometric features of input data; The graph convolution layer is used to extract global topological features of input data according to the local geometric features; The feature extraction layer and the graph convolution layer are both provided with multiple network levels, and the multimodal fusion layer is used to perform feature fusion on the calculation results of the multiple network levels; The deep feature extraction layer includes two fully connected layers for extracting deep features; The classification layer is used to output the classification results of each point.
5. The method according to claim 1, characterized in that The backbone and head of the first neural network are constructed based on the YOLO series algorithm, and the first neural network includes: a feature extraction network and a cross-level feature fusion network; The feature extraction network includes multiple levels from top to bottom, and different levels correspond to different resolutions; The cross-level feature fusion network comprises a plurality of stages, each stage having a level corresponding to at least part of the plurality of levels; In the first stage, there are multiple layers from bottom to top. Each layer performs feature fusion based on the output features of the previous layer in this stage and the output features of the same layer of the feature extraction network, and outputs the feature fusion result; In the second stage, which includes multiple levels from top to bottom, each level performs feature fusion based on the output features of the previous level in this stage, the output features of the previous level in the first stage, and the output features of the same level of the feature extraction network, and outputs the feature fusion results.
6. The method according to claim 5, characterized in that A C2f_faster module is provided in each level of the feature extraction network and the cross-level feature fusion network; In each level of the cross-level feature fusion network, a coordinate attention module is provided in the module that needs to be output; In the first stage, each level is provided with an upsampling module.
7. The method according to claim 6, characterized in that The coordinate attention module generates grid coordinate information for the input features, and performs mean pooling operations along the horizontal direction and the vertical direction for the grid coordinate information to obtain corresponding one-dimensional mean pooling results respectively; The one-dimensional mean pooling results corresponding to the horizontal direction and the vertical direction are spliced, and coordinate attention is generated according to the splicing results, and the number of channels is reduced according to the two-dimensional convolution; Through batch normalization and nonlinear activation function, regularization and activation processing are performed to generate the weights of coordinate attention.
8. The method according to claim 5, characterized in that The first neural network also includes: The small object segmentation head module includes multiple levels, and each level is connected to the corresponding level of the second stage.
9. A target positioning device based on image segmentation, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target positioning method based on image segmentation as described in any one of claims 1 to 8.
10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured as: a target positioning method based on image segmentation as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Multi-source remote sensing data adaptive fusion ground feature classification method
CN116543191A
Image and point cloud fused three-dimensional target detection method based on self-attention mechanism
CN118172535A
Three-dimensional lidar point cloud semantic segmentation method and apparatus based on deep learning
WO2024130776A1
Cited By
Fish form phenotype acquisition method and system based on multi-module cascaded 3D point cloud segmentation network
CN120564193A
