A spatial distance measurement method based on refined segmentation

By using a spatial ranging method based on fine segmentation and employing multiple models for image segmentation and ranging, the problems of inaccurate and slow detailed ranging in existing technologies are solved, enabling component-level ranging for each target and precise obstacle avoidance in three-dimensional space.

CN118351189BActive Publication Date: 2024-11-08SPEED SPACE TIME INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410757402.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-13
Publication Date
2024-11-08
Estimated Expiration
2044-06-13

AI Technical Summary

Technical Problem

Existing image segmentation methods suffer from problems such as inaccurate detailed ranging, slow speed, and lack of spatial features in spatial target ranging, making it difficult to achieve fine segmentation and ranging of each target.

Method used

A spatial ranging method based on fine segmentation is adopted, including steps such as image preprocessing, feature extraction and information fusion, mask prediction and hierarchical segmentation. Models such as Bidirectional Encoder Representations from Transformers (BERT), ResNet-50, Vision Transformer (ViT), Deformable DETR and MaskDINO are used for image segmentation and ranging.

Benefits of technology

It achieves component-level ranging for each target, improving obstacle avoidance accuracy and ranging speed in the field of autonomous driving, reducing costs, and enabling the acquisition of fine segmentation results in three-dimensional space.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118351189B_ABST
    Figure CN118351189B_ABST
Patent Text Reader

Abstract

The invention discloses a spatial distance measurement method based on refined segmentation, the steps are as follows: S1: preprocessing the image; S2: sorting the vocabulary; S3: transferring the image to the image detection model for feature extraction and information fusion, decoupling the foreground and background and performing mask prediction to generate the foreground mask and the background mask; performing retrieval, and finally performing hierarchical segmentation and class-aware component segmentation to obtain a segmented image; S4: selecting the target category to be measured for target distance measurement, obtaining the distance from the target mask to the image acquisition device, that is, the distance from the target to the image acquisition device; S5: calculating the maximum distance and the minimum distance of the target in the three-dimensional space, thereby completing the spatial distance measurement of the target and obtaining the stereoscopic distance of the target in the three-dimensional space. The method can obtain a refined segmentation result for each target in the three-dimensional space, thereby performing distance measurement for each component, and then obtaining the stereoscopic distance of the target in the three-dimensional space from the distance set.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a spatial distance measurement method based on refined segmentation. Background Art

[0002] Image segmentation refers to dividing an image into several non-intersecting regions based on features such as grayscale, color, spatial texture, and geometric shape, so that these features show consistency or similarity in the same region, and show obvious differences between different regions. Simply put, it is to separate the target from the background in an image. For grayscale images, pixels inside the region generally have grayscale similarity, while there is generally grayscale discontinuity at the boundary of the region. Regarding image segmentation technology, due to the importance and difficulty of the problem itself, there is no universal and perfect image segmentation method, but there is basically a consensus on the general rules of image segmentation, and a considerable number of research results and methods have been produced. There are currently two commonly used methods. One is to measure the distance based on the results of a single target in the image. The other is to obtain the distance measurement results based on professional acquisition equipment (lidar, rangefinder).

[0003] The traditional segmentation method currently used is a method of image segmentation that uses knowledge in digital image processing, topology, mathematics, etc. The basic idea of ​​the threshold-based segmentation method is to calculate one or more grayscale thresholds based on the grayscale features of the image, and compare the grayscale value of each pixel in the image with the threshold, and finally classify the pixels into appropriate categories based on the comparison results. Therefore, the most critical step of this method is to solve the optimal grayscale threshold according to a certain criterion function. The threshold method is particularly suitable for images where the target and background occupy different grayscale ranges.

[0004] If the image has only two categories, target and background, then only one threshold needs to be selected for segmentation, and this method is called single threshold segmentation; but if there are multiple targets to be extracted in the image, the segmentation of a single threshold will result in crops. In this case, multiple thresholds need to be selected to separate each target, and this segmentation method is called multi-threshold segmentation. The region-based segmentation method is a segmentation technology based on direct search for regions. There are two basic forms of region-based extraction methods: one is region growing, starting from a single pixel, gradually merging to form the required segmentation region; the other is starting from the global, gradually cutting to the required segmentation region. Region growing, region growing starts with a group of seed pixels representing different growth regions, then merges the pixels that meet the conditions in the neighborhood of the seed pixel into the growth region represented by the seed pixel, and continues the merging process with the newly added pixels as new seed pixels until no new pixels that meet the conditions are found. Region splitting and merging can be said to be the inverse process of region growing. Starting from the entire image, it continuously splits to obtain each sub-region, and then merges the foreground region to obtain the foreground target that needs to be segmented, thereby achieving target extraction. In practical applications, the region growing algorithm and the region splitting and merging algorithm are usually used in combination. This type of algorithm is ideal for segmenting complex scenes defined by certain complex objects or for segmenting certain natural scenes, and other similar image segmentation effects with insufficient prior knowledge. Image segmentation algorithms based on edge detection attempt to detect the edges of different regions. Usually, the grayscale values ​​of pixels on the boundaries of different regions change dramatically. If the image is transformed from the spatial domain to the frequency domain through Fourier transform, the edge corresponds to the high-frequency part. This is a very simple edge detection algorithm. Segmentation based on deep learning includes feature encoder based. In the field of feature extraction, VGGnet and ResNet are two very effective methods. Among them, the advantages and disadvantages of the threshold segmentation method are: simple calculation and high efficiency; only the characteristics of the grayscale value of the pixel itself are considered, and the spatial characteristics are generally not considered, so it is sensitive to noise and has low robustness.

[0005] The region-based image segmentation method mainly relies on the three problems that the region-based region growing algorithm needs to solve: (1) Select or determine a set of seed pixels that can correctly represent the required region; (2) Determine the criteria for including adjacent pixels in the growing process; (3) Specify the conditions or rules for stopping the growing process. Therefore, the advantages and disadvantages of the region splitting and merging algorithm are: (1) Good segmentation effect for complex images; (2) The algorithm is complex and the calculation is large; (3) Splitting may break the boundaries of strange regions.

[0006] Advantages and disadvantages of edge detection: (1) Accurate edge positioning; (2) Fast speed; (3) Unable to guarantee the continuity and closure of the edge; (4) There are a large number of fragmented edges in the high-detail area, which makes it difficult to form a large area, but it is not suitable to divide the high-detail area into small fragments.

[0007] Advantages and disadvantages of VGGNet: (1) Since the number of parameters is mainly concentrated in the last three FCs, deepening the network will not cause parameter explosion problems; (2) The receptive field of multiple small-core convolutional layers is equivalent to that of a large-core convolutional layer (three 3x3 layers are equivalent to one 7x7 layer), but the number of parameters is far less than that of the large-core convolutional layer and the number of nonlinear operations is also more than the latter, which makes it have stronger learning ability; (3) VGG occupies more memory (140M) due to the large number of layers and the large number of parameters in the last three fully connected layers.

[0008] Advantages and disadvantages of ResNet: (1) It introduces a new network structure (residual learning module) and forms a new network structure, which can make the network as deep as possible; (2) It enables the feedforward / feedback propagation algorithm to proceed smoothly and the structure is simpler; (3) The increase of identity mapping will basically not reduce the performance of the network; (4) It constructively solves the problem that the deeper the network training, the higher the error and the more obvious the gradient disappears; (5) Since ResNet has a large number of layers, the training time required is also longer than that of ordinary networks.

[0009] Advantages and disadvantages of acquisition technology: (1) Ordinary color cameras have better versatility and resolution, and can better obtain detailed images; (2) Ordinary color cameras have lower costs and are easier to maintain and install; (3) Ordinary color cameras cannot obtain depth and distance information.

[0010] Chinese patent document CN115116026A discloses an automatic tracking method and system for a logistics handling robot. The Mask RCNN segmentation method adopted in the technical solution is a traditional general segmentation method. The segmentation effect is holistic segmentation, which can only achieve the segmentation of the object as a whole. For example, for the segmentation of a person, only the segmentation effect of the whole person can be obtained.

[0011] Chinese patent document CN116168103A discloses a method for fusing three-dimensional laser radar ranging and intensity information and expressing them in images. The acquisition equipment used in this technical solution is very different from our invention. This technical solution uses laser radar, which has a high cost and high maintenance and installation costs. In addition, this technical solution focuses on fusing radar ranging information and intensity information, rather than segmenting and measuring the distance of the image. Summary of the invention

[0012] The technical problem to be solved by the present invention is to provide a spatial ranging method based on refined segmentation, which can solve the problems of inaccurate detail ranging, slow ranging speed, lack of spatial features, etc. in the ranging of space targets. The method can obtain a refined segmentation result for each target in three-dimensional space, and measure the distance of each component based on the refined segmentation result, so as to obtain the stereoscopic distance of the target in three-dimensional space in a distance set.

[0013] In order to solve the above technical problems, the technical solution adopted by the present invention is: the spatial ranging method based on refined segmentation specifically comprises the following steps:

[0014] S1: preprocess the received image;

[0015] S2: Organize the vocabulary based on the preprocessed images;

[0016] S3: The image is passed into the image detection model for feature extraction and information fusion, and then the foreground and background are decoupled and mask prediction is performed to generate foreground masks and background masks; then retrieval is performed, and finally hierarchical segmentation and class-aware component segmentation are performed to obtain a segmented image;

[0017] S4: According to the acquired segmented image, select the target category to be measured for target ranging, and obtain the distance from the target mask to the image acquisition device, that is, the distance from the target to the image acquisition device;

[0018] S5: According to step S4, the distances of all masks of the target are obtained, and the maximum distance and the minimum distance of the target in the three-dimensional space are calculated, thereby completing the spatial distance measurement of the target and obtaining the stereoscopic distance of the target in the three-dimensional space.

[0019] By adopting the above technical solution, the method performs detailed segmentation on the target, and according to the result of the detailed segmentation, realizes the component-level distance measurement of each target, so as to estimate the distance set of the target from the camera in the three-dimensional space, which is conducive to the accurate obstacle avoidance in the field of autonomous driving. By detecting and segmenting the image very carefully, a more detailed segmentation than the overall segmentation can be obtained. For example, for the segmentation of a person, the segmentation results of various parts of the human body: hair, face, arms, palms, upper body, feet, and legs can be obtained; the segmentation results are more detailed, so in the field of autonomous driving, with this detailed segmentation result, the distance measurement can be performed on each part, which is conducive to the avoidance scheme of autonomous driving and is conducive to improving the success rate of the scene of overtaking pedestrians on narrow roads. The method can obtain a fine segmentation result for each target in three-dimensional space, and measure the distance of each component based on the fine segmentation result, so as to obtain the distance set. Knowing the stereoscopic distance of the target in three-dimensional space is conducive to the field of autonomous driving, based on the spatial distance set of each target, which can help the more refined operation of autonomous driving.

[0020] Preferably, the preprocessing in step S1 includes color correction, noise reduction and data enhancement, and the specific steps are:

[0021] S11 color correction: First, the input image is histogram averaged, and the histogram of the input image is mapped or contrast stretched so that the histogram of the output image is evenly distributed in the entire chromaticity range, thereby enhancing the image contrast;

[0022] S12 noise reduction: uses wavelet transform technology to decompose the image into sub-bands of different frequency ranges;

[0023] S13 Data Enhancement: Enhance the quality of samples through image processing technology, correct the direction of images through rotation, and improve the success rate in the subsequent segmentation process. Data enhancement mainly uses image processing technology to enhance the quality of samples and obtain a more robust model.

[0024] Preferably, in step S11, the contrast stretching is performed by finding a pair of extreme values, namely, the 2% and 98% percentiles of the image, and then stretching the extreme value range to the entire chromaticity range, so that darker pixels are made darker and brighter pixels are made brighter, thereby enhancing the image contrast; the formula is:

[0025] New_Pixel = (Old_Pixel - min)*(255 / (max - min));

[0026] Among them, max is the maximum grayscale value of the original image, min is the minimum grayscale value of the original image, Old_Pixel is the original pixel value, and New_Pixel is the new pixel value.

[0027] Preferably, in step S12, the image is decomposed into four sub-bands, namely, LL low-frequency sub-band (low-frequency part), LH horizontal high-frequency sub-band (horizontal high-frequency part), HL vertical high-frequency sub-band (vertical high-frequency part) and HH diagonal high-frequency sub-band (diagonal high-frequency part); the wavelet coefficients of the LL low-frequency sub-band (low-frequency part), LH horizontal high-frequency sub-band (horizontal high-frequency part), HL vertical high-frequency sub-band (vertical high-frequency part) and HH diagonal high-frequency sub-band (diagonal high-frequency part) together constitute the image after wavelet transformation, and then the noise is removed according to the frequency selectivity, so as to realize the denoising of the image. The wavelet transformation adopts discrete wavelet transform (Discrete Wavelet Transform, DWT). Under discrete wavelet transform, the wavelet coefficient I' of image I is expressed as:

[0028] I' = ∑ I(x,y) * ψ(t - x, s - y) dxdy;

[0029] Among them, I is a two-dimensional input image or signal, ψ represents the wavelet function operation, which has localized and zero-mean characteristics; (x, y) represents the spatial coordinates in the image, (t, s) are the parameters of the wavelet function translation operation, t represents the translation operation, which is used to control the position of the wavelet on the image, and s represents the scale parameter, which is used to control the frequency content of the wavelet.

[0030] Preferably, in step S3, feature extraction and information fusion are first performed, and then a decoder is used to decouple the foreground and background and perform mask prediction to generate a foreground mask and a background mask; then the candidate box and the mask are used for retrieval, and finally hierarchical segmentation is performed according to component perception, and then class-aware component segmentation is performed to obtain a segmented image; the specific steps are:

[0031] S31: Extract features and fuse information from the incoming image and text information. Use the pre-trained model (Bidirectional Encoder Representations from Transformers, BERT) to extract the features of text prompts. Use the 50-layer residual network (ResNet-50) and the vision transformer (ViT) as the basic architecture for image encoding. The multi-scale features of the image are obtained as F v ;

[0032] S32: Extracting text-guided visual features using a bidirectional cross-attention mechanism F t2v and Image-guided text features F v2t , and then the attention feature is connected with the text feature through residual connection F t and image multi-scale features F v Integration, the formula is:

[0033] ;

[0034] ;

[0035] in, Would - Xattn represents the bidirectional cross attention mechanism, F t Represents text features, F v Represents the multi-scale features of the image, represents the resulting image features, Represents the result text features;

[0036] S33: Two separate decoders are used to decouple the foreground mask prediction and the background mask prediction. The foreground mask prediction adopts a deformable convolutional network model (Deformable DETR). Deformable DETR can introduce a deformable convolutional network to make the model focus more on local features rather than the entire image, thereby greatly improving the performance in dealing with long-tail distribution and small target problems; the main strategy of DETR is binary matching, and its formula is:

[0037] L = λ_bb L_box + L_class;

[0038] Among them, L_box represents the loss between the predicted box position and the real box position (using L1 loss), L_class represents the cross entropy loss between the predicted category and the real category; λ_bb is a hyperparameter that controls the weight between the two losses;

[0039] The mask prediction of the background adopts the MaskDINO architecture, uses a one-to-one Hungarian matching method, and does not consider the loss value of the box for prediction;

[0040] S34: Use text-based prompt box and mask retrieval, setting the feature to E , the test class is C test ; First, calculate the probability using standard methods p 1 ( E , C test )= P ( C test|E ), and compute the mask pooling features of M from the visual encoder V of the contrastive language-image pre-trained CLIP text features as E CLIP , obtained through MaskPooling mask pooling operation:

[0041] E CLIP = MaskPooling( M , V (I));

[0042] in, M is a set of masks, V (I) is the feature obtained by the visual encoder after contrastive language-image pre-training CLIP for image I;

[0043] Then calculate CLIP logits p2( E , C test ) =P ( C test | E CLIP ) as a contrastive language-image pre-trained CLIP text features and E CLIP The similarity between them, and finally the final prediction result p final Merge into:

[0044] ;

[0045] in, λ It is a balancing factor; C test Is a collection of test classes, E CLIP is the mask pool feature;

[0046] S35: Using component-aware hierarchical segmentation, the instance class name and the component name are merged and connected as labels; then, based on the results of combining instance-level segmentation and part segmentation, instance segmentation based on part awareness is implemented; then, the components are grouped to obtain the hierarchical structure of the components;

[0047] S36: Use the segmentation model to perform class-aware component segmentation. Specifically, by combining the semantic output with the class-aware mask output by the segmentation model, first define the semantic level mask as A , and its class probability P A , A k The semantic level mask is A of k Widget subclass for masks generated by segmentation models S , calculate the mask Yes ∈ S Relative to class j The class probability is:

[0048] ;

[0049] Among them, | A k ∩ S i |For mask A k and S i The intersection area of ​​​​the semantic output is combined with the segmentation model; i is a constant.

[0050] Preferably, the deformable convolutional network model Deformable DETR in the foreground mask in step S33 adopts the DINO architecture and adds denoising. When calculating the loss, the teacher assistant implements a simultaneous optimization strategy (simultaneous Optimization through Teacher Assistant, simOTA) to perform a many-to-one match between the generated prediction box and the true value, and uses box-based non-maximum suppression NMS to delete duplicate predictions.

[0051] Preferably, the specific steps of step S4 are:

[0052] S41: Select the target category to be measured, obtain the mask of the target, and combine the detailed segmentation results of all target categories obtained in step S3. Since each target is composed of multiple masks, all mask images contained in each target T are measured, and then one of the masks T1 is selected;

[0053] S42: Frame the range where the mask T1 selected in step S41 is located, and obtain the pixel side length P1 and the pixel side length P2 in the frame;

[0054] S43: Calculate the actual distance D from the object to the image acquisition device according to the pixel side length P1 or the pixel side length P2, the pre-determined focal length of the image acquisition device and the actual distance of the object, using the formula:

[0055] D = (W * F) / P1;

[0056] Where: D is the actual distance from the object to the image acquisition device, W is the actual width of the object, F is the focal length of the image acquisition device, which is measured in advance by a real object at a known distance; P1 is the pixel side length of the object in the image.

[0057] Preferably, the specific steps of step S5 are: according to the distances of all masks of the target obtained in step S4, a distance set D={D1, D 2, …, D N}, get the maximum distance D of the target in three-dimensional space from the distance set max and the minimum distance D min , the formula is:

[0058] D max = MAX(D1, D2, …, D N );

[0059] D min = MIN(D1, D2, …, D N );

[0060] Among them, D max is the maximum distance, D min is the minimum distance.

[0061] Preferably, the vocabulary in step S2 includes an open vocabulary of the field to be detected and a custom vocabulary for the field of autonomous driving.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] (1) This method performs detailed segmentation on the target and achieves component-level distance measurement for each target based on the result of the detailed segmentation, thereby estimating the set of distances of the target from the camera in three-dimensional space, which is beneficial for accurate obstacle avoidance in the field of autonomous driving. Compared with traditional methods, this method can achieve better segmentation effect. Traditional methods can only achieve segmentation of a single target and cannot achieve fine segmentation of the target.

[0064] (2) The present invention can also obtain a distance set based on fine segmentation, which has a better ranging effect than the single distance of the traditional ranging method;

[0065] (3) The present invention uses a common color camera to achieve segmentation, which can better control costs compared with the existing laser ranging method. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flow chart of the spatial distance measurement method based on refined segmentation of the present invention;

[0067] FIG. 2 (a) is an original image to be processed by the spatial distance measurement method based on refined segmentation of the present invention;

[0068] Figure 2 (b) is the result of the traditional segmentation method;

[0069] FIG. 2( c ) is a diagram showing the result of the segmentation method of the spatial distance measurement method based on refined segmentation according to the present invention. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiment diagram of the present invention.

[0071] Example: Figure 1 As shown, the spatial distance measurement method based on refined segmentation specifically includes the following steps:

[0072] S1: The image acquisition device in this embodiment is a camera, and the received image is first preprocessed; the preprocessing in step S1 includes color correction, noise reduction and data enhancement, and the specific steps are:

[0073] S11 color correction: First, the input image is histogram averaged, and the histogram of the input image is mapped or contrast stretched so that the histogram of the output image is evenly distributed in the entire chromaticity range, thereby enhancing the image contrast;

[0074] In step S11, the contrast stretching is performed by finding a pair of extreme values, namely the 2% and 98% percentiles of the image, and then stretching the extreme value range to the entire chromaticity range, so that darker pixels are made darker and brighter pixels are made brighter, thereby enhancing the image contrast; the formula is:

[0075] New_Pixel = (Old_Pixel - min)*(255 / (max - min));

[0076] Where max is the maximum grayscale value of the original image, min is the minimum grayscale value of the original image, Old_Pixel is the original pixel value, and New_Pixel is the new pixel value;

[0077] S12 noise reduction: uses wavelet transform technology to decompose the image into sub-bands of different frequency ranges;

[0078] In step S12, the image is decomposed into four sub-bands, namely, LL low-frequency sub-band (low-frequency part), LH horizontal high-frequency sub-band (horizontal high-frequency part), HL vertical high-frequency sub-band (vertical high-frequency part) and HH diagonal high-frequency sub-band (diagonal high-frequency part); the wavelet coefficients of the LL low-frequency sub-band (low-frequency part), LH horizontal high-frequency sub-band (horizontal high-frequency part), HL vertical high-frequency sub-band (vertical high-frequency part) and HH diagonal high-frequency sub-band (diagonal high-frequency part) together constitute the image after wavelet transformation, and then the noise is removed according to the frequency selectivity, so as to realize the denoising of the image. The wavelet transformation adopts discrete wavelet transform (Discrete Wavelet Transform, DWT). Under discrete wavelet transform, the wavelet coefficient I' of image I is expressed as:

[0079] I' = ∑ I(x,y) * ψ(t - x, s - y) dxdy;

[0080] Where I is a two-dimensional input image or signal, ψ represents the wavelet function operation, which has localized and zero-mean characteristics; (x, y) represents the spatial coordinates in the image, (t, s) are the parameters of the wavelet function translation operation, t represents the translation operation, which is used to control the position of the wavelet on the image, and s represents the scale parameter, which is used to control the frequency content of the wavelet;

[0081] S13 Data Enhancement: Enhance the quality of samples through image processing technology, correct the direction of images through rotation, and improve the success rate of subsequent segmentation processes; Data enhancement mainly uses image processing technology to enhance the quality of samples and obtain a more robust model;

[0082] S2: Arranging a vocabulary list according to the preprocessed image; the vocabulary list in step S2 includes an open vocabulary list of the field to be detected and a vocabulary list for the field of autonomous driving;

[0083] S3: The image is passed into the image detection model for feature extraction and information fusion, and then the foreground and background are decoupled and mask prediction is performed to generate foreground mask and background mask; then retrieval is performed, and finally hierarchical segmentation and class-aware component segmentation are performed to obtain a segmented image;

[0084] In step S3, feature extraction and information fusion are first performed, and then a decoder is used to decouple the foreground and background and perform mask prediction to generate a foreground mask and a background mask; then the candidate box and the mask are used for retrieval, and finally hierarchical segmentation is performed according to component perception, and then class-aware component segmentation is performed to obtain a segmented image; the specific steps are:

[0085] S31: Extract features and fuse information from the incoming image and text information. Use the pre-trained model (Bidirectional Encoder Representations from Transformers, BERT) to extract the features of text prompts. Use the 50-layer residual network ResNet-50 and the vision transformer (ViT) as the basic architecture for image encoding. The multi-scale features of the image are obtained as F v ;

[0086] S32: Extracting text-guided visual features using a bidirectional cross-attention mechanism F t2v and Image-guided text features F v2t , and then the attention feature is connected with the text feature through residual connection F t and image multi-scale features F v Integration, the formula is:

[0087] ;

[0088] ;

[0089] in, Would - Xattn represents the bidirectional cross attention mechanism, F t Represents text features, F v Represents the multi-scale features of the image, represents the resulting image features, Represents the result text features;

[0090] S33: Two separate decoders are used to decouple the foreground mask prediction and the background mask prediction. The foreground mask prediction adopts a deformable convolutional network model (Deformable DETR). Deformable DETR can introduce a deformable convolutional network to make the model focus more on local features rather than the entire image, thereby greatly improving the performance in dealing with long-tail distribution and small target problems; the main strategy of DETR is binary matching, and its formula is:

[0091] L = λ_bb L_box + L_class;

[0092] Wherein, L_box represents the loss between the predicted box position and the real box position (using L1 loss), L_class represents the cross entropy loss between the predicted category and the real category; λ_bb is a hyperparameter that controls the weight between the two losses, which is usually set to 5, and is set to 5 in this embodiment; the deformable convolutional network model Deformable DETR in the foreground mask in the step S33 adopts the DINO architecture and adds denoising, and when calculating the loss, the teacher assistant is used to implement the simultaneous optimization strategy (simultaneous optimization through teacher assistant, simOTA) to perform a many-to-one match between the generated predicted box and the real value, and the box-based non-maximum suppression NMS is used to delete duplicate predictions;

[0093] The mask prediction of the background adopts the MaskDINO architecture, uses a one-to-one Hungarian matching method, and does not consider the loss value of the box for prediction;

[0094] S34: Use text-based prompt box and mask retrieval, setting the feature to E , the test class is C test ; First, calculate the probability using standard methods p 1 ( E , C test ) = P( C test|E), and compute the mask pooling features of M from the visual encoder V of the contrastive language-image pre-trained CLIP text features as E CLIP , obtained through MaskPooling mask pooling operation:

[0095] E CLIP = MaskPooling( M , V (I));

[0096] in, M is a set of masks, V (I) is the feature obtained by the visual encoder after contrastive language-image pre-training CLIP for image I;

[0097] Then calculate CLIP logits p2( E , C test ) = P ( C test | E CLIP ) as a contrastive language-image pre-trained CLIP text features and E CLIP The similarity between them, and finally the final prediction result p final Merge into:

[0098] ;

[0099] in, λ It is a balancing factor; C test Is a collection of test classes, E CLIP is the mask pool feature;

[0100] S35: Using component-aware hierarchical segmentation, the instance class name and the component name are merged and connected as labels; then, based on the results of combining instance-level segmentation and part segmentation, instance segmentation based on part awareness is implemented; then, the components are grouped to obtain the hierarchical structure of the components;

[0101] S36: Use the segmentation model to perform class-aware component segmentation. Specifically, by combining the semantic output with the class-aware mask output by the segmentation model, first define the semantic level mask as A , and its class probability P A , A k The semantic level mask is A of kWidget subclass for masks generated by segmentation models S , calculate the mask Yes ∈ S Relative to class j The class probability is:

[0102] ;

[0103] Among them, | A k ∩ S i |For mask A k and S i The intersection area of ​​​​the semantic output is combined with the segmentation model; i is a constant;

[0104] S4: According to the acquired segmented image, select the target category to be measured for target ranging, and obtain the distance from the target mask to the image acquisition device, that is, the distance from the target to the image acquisition device;

[0105] The specific steps of step S4 are:

[0106] S41: Select the target category to be measured, obtain the mask of the target, and combine the detailed segmentation results of all target categories obtained in step S3. Since each target is composed of multiple masks, all mask images contained in each target T are measured, and then one of the masks T1 is selected;

[0107] S42: Frame the range where the mask T1 selected in step S41 is located, and obtain the pixel side length P1 and the pixel side length P2 in the frame;

[0108] S43: Calculate the actual distance D from the object to the image acquisition device according to the pixel side length P1 or the pixel side length P2, the pre-determined focal length of the image acquisition device and the actual distance of the object, using the formula:

[0109] D = (W * F) / P1;

[0110] Where: D is the actual distance from the object to the image acquisition device, W is the actual width of the object, F is the focal length of the image acquisition device, which is pre-measured by a real object at a known distance; P1 is the pixel side length of the object in the image;

[0111] S5: According to step S4, the distances of all masks of the target are obtained, and the maximum distance and the minimum distance of the target in the three-dimensional space are calculated, so as to complete the spatial distance measurement of the target and obtain the stereoscopic distance of the target in the three-dimensional space; the specific steps of step S5 are: according to step S4, the distances of all masks of the target are obtained to form a distance set D={D1, D 2, …,D N}, get the maximum distance D of the target in three-dimensional space from the distance set max and the minimum distance D min , the formula is:

[0112] D max = MAX(D1, D2, …, D N );

[0113] D min = MIN(D1, D2, …, D N );

[0114] Among them, D max is the maximum distance, D min is the minimum distance.

[0115] As shown in Figure 2 (a) to Figure 2 (c), the method of the present invention can better achieve segmentation, and for each target in the picture, a detailed segmentation result can be obtained, such as: a person can be segmented into head, neck, palm, arm and other parts. The technical solution of the present invention can measure the distance of each part based on the result of the segmentation of each part. With the detailed distance measurement results, the autonomous driving technology can make an avoidance plan based on the distance measurement results of each part of the human body.

[0116] The present invention can be applied in driverless cars and deployed at the front end of the vehicle to measure the distance of fixed-size objects in front. Most vehicles now have reversing radars. By using the present invention, it is possible to use ordinary cameras with lower costs to achieve refined segmentation, and obtain the three-dimensional distance of each target based on the segmentation results. The present invention mainly includes two steps: segmentation and ranging. The image obtained by the camera is segmented and passed into the segmentation model, and the distance is measured according to the results of the segmentation model.

[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A spatial distance measurement method based on refined segmentation, characterized in that: The specific steps include: S1: preprocess the received image; S2: Organize the vocabulary based on the preprocessed images; S3: The image is passed into the image detection model for feature extraction and information fusion, then the foreground and background are decoupled and mask prediction is performed to generate foreground masks and background masks, and then retrieval is performed, and finally hierarchical segmentation and class-aware component segmentation are performed to obtain a segmented image; S4: According to the acquired segmented image, select the target category to be measured for target ranging, and obtain the distance from the target mask to the image acquisition device, that is, the distance from the target to the image acquisition device; S5: Obtain the distances of all masks of the target according to step S4, calculate the maximum distance and the minimum distance of the target in the three-dimensional space, thereby completing the spatial distance measurement of the target and obtaining the stereoscopic distance of the target in the three-dimensional space; The specific steps of step S4 are: S41: Select the target category to be measured, obtain the mask of the target, combine the segmentation results of all target categories obtained in step S3, and measure the distance of all mask images contained in each target T, because each target is composed of multiple masks, and then select one of the masks T1; S42: Frame the range where the mask T1 selected in step S41 is located, and obtain the pixel side length P1 and the pixel side length P2 in the frame; S43: Calculate the actual distance D from the object to the image acquisition device according to the pixel side length P1 or the pixel side length P2, the pre-determined focal length of the image acquisition device and the actual distance of the object, using the formula: D = (W*F) / P1; Where: D is the actual distance from the object to the image acquisition device, W is the actual width of the object, F is the focal length of the image acquisition device, which is measured in advance by a real object at a known distance; P1 is the pixel side length of the object in the image.

2. The spatial distance measurement method based on refined segmentation according to claim 1, characterized in that: The preprocessing in step S1 includes color correction, noise reduction and data enhancement, and the specific steps are as follows: S11 color correction: First, the input image is histogram averaged, and the histogram of the input image is mapped or contrast stretched so that the histogram of the output image is evenly distributed in the entire chromaticity range, thereby enhancing the image contrast; S12 noise reduction: uses wavelet transform technology to decompose the image into sub-bands of different frequency ranges; S13 Data Enhancement: Enhance the quality of samples through image processing technology, correct the direction of the image through rotation, and improve the success rate of subsequent segmentation processes.

3. The spatial distance measurement method based on refined segmentation according to claim 2, characterized in that: In step S11, the contrast stretching is performed by finding a pair of extreme values ​​and then stretching the range of the extreme values ​​to the entire chromaticity range, and the formula is: New_Pixel = (Old_Pixel-min)*(255 / (max-min)); Among them, max is the maximum grayscale value of the original image, min is the minimum grayscale value of the original image, Old_Pixel is the original pixel value, and New_Pixel is the new pixel value.

4. The spatial distance measurement method based on refined segmentation according to claim 2, characterized in that: In step S12, the image is decomposed into four sub-bands, namely, LL low-frequency sub-band, LH horizontal high-frequency sub-band, HL vertical high-frequency sub-band and HH diagonal high-frequency sub-band; the wavelet coefficients of the LL low-frequency sub-band, LH horizontal high-frequency sub-band, HL vertical high-frequency sub-band and HH diagonal high-frequency sub-band together constitute the image after wavelet transformation, and then the noise is removed according to the frequency, so as to realize the denoising of the image. The wavelet transformation adopts discrete wavelet transformation. Under discrete wavelet transformation, the wavelet coefficient I' of the image I is expressed as: I'=∑I(x,y)*ψ(tx,sy)dxdy; Among them, I is a two-dimensional input image or signal, ψ represents the wavelet function operation, which has localized and zero-mean characteristics; (x, y) represents the spatial coordinates in the image, (t, s) are the parameters of the wavelet function translation operation, t represents the translation operation, which is used to control the position of the wavelet on the image, and s represents the scale parameter, which is used to control the frequency content of the wavelet.

5. The spatial distance measurement method based on refined segmentation according to claim 2, characterized in that: In step S3, feature extraction and information fusion are first performed, and then a decoder is used to decouple the foreground and background and perform mask prediction to generate a foreground mask and a background mask; then the candidate box and the mask are used for retrieval, and finally hierarchical segmentation is performed according to component perception, and then class-aware component segmentation is performed to obtain a segmented image; the specific steps are: S31: Extract features and fuse information from the incoming image and text information, use the pre-trained model to extract the features of the text prompt, use a 50-layer residual network and a visual transformer as the basic architecture for image encoding, and obtain the multi-scale features of the image as F v ; S32: Extracting text-guided visual features using a bidirectional cross-attention mechanism F t2v and image-guided text features F v2t , and then the attention feature is connected with the text feature F through residual connection t and image multi-scale features F v Integration, the formula is: F t2v ,F v2t =Bi-Xattn(F v ,F t ); (F′ v ,F′ t )=(F t +F t2v ,F t +F v2t ); Among them, Bi-Xattn represents the bidirectional cross attention mechanism, F t represents the text feature, F v Represents the multi-scale features of the image, F′ v Denotes the resulting image features, F′ t Represents the result text features; S33: using two separate decoders to decouple foreground mask prediction and background mask prediction, respectively, the foreground mask prediction adopts a deformable convolutional network model, and its strategy is binary matching, the formula is: L = λ_bb L_box + L_class; Among them, L_box represents the loss between the predicted box position and the true box position, L_class represents the cross entropy loss between the predicted category and the true category; λ_bb is a hyperparameter that controls the weight between the two losses; the mask prediction of the background adopts the MaskDINO architecture, adopts a one-to-one Hungarian matching method, and does not consider the loss value of the box for prediction; S34: Using text-based prompts and mask retrieval, setting the feature to E and the test class to C test ; First, calculate the probability p1(E,C test )=P(C test|E ), and compute the mask pooling features of M as E from the visual encoder V of the contrastive language-image pre-trained CLIP text features CLIP , obtained through MaskPooling mask pooling operation: E CLIP =MaskPooling(M,V(I)); Where M is the set of masks, V(I) is the features of image I obtained by the visual encoder after contrastive language-image pre-training CLIP; Then calculate CLIP logits p2(E,C test )=P(C test |E CLIP ) as a contrastive language-image pre-trained CLIP text features and E CLIP The similarity between them, and finally the final prediction result p final Merge into: p final (E,C test )∝p1(E,C test ) λ p2(E,C test ) 1-λ ; Among them, λ is a balancing factor, C test is a set of test classes, E CLIP is the mask pool feature; S35: Using component-aware hierarchical segmentation, the instance class name and the component name are merged and connected as labels; then, based on the results of combining instance-level segmentation and part segmentation, instance segmentation based on part awareness is implemented; then, the components are grouped to obtain the hierarchical structure of the components; S36: Use the segmentation model to perform class-aware component segmentation. Specifically, by combining the semantic output and the class-aware mask output by the segmentation model, first define the semantic level mask as A and its class probability P A , A k is the k-component subclass with semantic-level mask A. For the mask S generated by the segmentation model, the class probability of mask Si∈S relative to class j is calculated as: Among them, |A k ∩S i | is mask A k With S i The intersection area of ​​, that is, combining the semantic output with the segmentation model; i is a constant.

6. The spatial distance measurement method based on refined segmentation according to claim 5, characterized in that: The deformable convolutional network model in the foreground mask in step S33 adopts the DINO architecture and adds denoising. When calculating the loss, a teacher assistant is used to implement a synchronous optimization strategy to perform a many-to-one matching between the generated prediction box and the true value, and a box-based non-maximum suppression NMS is used to delete duplicate predictions.

7. The spatial distance measurement method based on refined segmentation according to claim 5, characterized in that: The specific steps of step S5 are: according to the distances of all masks of the target obtained in step S4, a distance set D={D1,D 2, …,D N }, get the maximum distance D of the target in three-dimensional space from the distance set max and the minimum distance D min , the formula is: D max =MAX(D1,D2,…,D N ); D min =MIN(D1,D2,…,D N ); Among them, D max is the maximum distance, D min is the minimum distance.

8. The spatial distance measurement method based on refined segmentation according to claim 5, characterized in that: The vocabulary in step S2 includes an open vocabulary of the field to be detected and a vocabulary for the field of autonomous driving.