Texture similarity detection method and device, medium and equipment

Abstract

The invention discloses a texture similarity detection method and device, a medium and equipment, and relates to the technical field of pose estimation, and the method comprises the steps: obtaining all reference image feature vectors of a target offline; inputting the query image of the object to be detected into a trained texture similarity detection model for texture similarity detection to obtain a texture feature vector of the reference image; the texture similarity detection model comprises a multi-scale feature extraction module, a multi-scale feature processing module, a cross-scale attention fusion module and a feature projection head which are connected in sequence; and performing dot product on the texture feature vector and each reference image feature vector to determine a plurality of dot product results, sorting the plurality of dot product results from large to small, and taking the plurality of sorted dot product results as query results of texture similar images.

Description

Technical Field

[0001] This invention relates to the field of pose estimation, and in particular to a texture similarity detection method, apparatus, medium, and device. Background Technology

[0002] Texture similarity detection technology is trending towards a shift from traditional hand-crafted features to deep neural network features. The core of texture similarity detection is quantifying the degree of similarity between two images in terms of texture. This relies on two main steps: texture feature extraction and similarity measurement. The key challenges lie in designing feature descriptors robust to changes in lighting, rotation, and scale, and in striking a balance between computational efficiency and feature discriminative power. Traditional image processing methods for texture similarity detection mainly include those based on Gabor wavelet transform, LBP (Local Binary Pattern), and GLCM (Gray-Level Co-occurrence Matrix). These traditional approaches use hand-designed features to characterize textures. Deep learning methods based on Siamese networks directly map the input to a new feature space and calculate its similarity. This method is suitable for items with unique textures, such as tea cakes and marble.

[0003] However, traditional methods suffer from limitations in computational efficiency and applicability to various scenarios. While deep learning methods demonstrate strong potential in adaptability and accuracy in complex scenes, training a robust and powerful texture similarity model typically requires large-scale, high-quality labeled datasets. Furthermore, texture similarity itself is a subjective and continuous concept, making it extremely difficult and expensive to obtain accurate "similarity score" annotations. In short, in uncontrolled environments, factors such as occlusion, shadows, noise, and viewpoint changes can significantly interfere with the appearance of textures, posing challenges to accurate texture similarity detection. Summary of the Invention

[0004] This invention provides a texture similarity detection method, apparatus, medium, and device to solve the aforementioned problems in the prior art, namely, how to improve the accuracy of texture similarity detection in the prior art. This invention provides a texture similarity detection method, which includes: Obtain the feature vectors of all offline reference images of the target; The query image of the object to be tested is input into a trained texture similarity detection model to obtain the texture feature vector of the reference image. The texture similarity detection model includes a multi-scale feature extraction module, a multi-scale feature processing module, a cross-scale attention fusion module, and a feature projection head connected in sequence. Specifically, the multi-scale feature extraction module extracts features at various scales of the query image to determine a multi-scale feature map; the multi-scale feature processing module processes the multi-scale feature map to determine attention features at different scales; the cross-scale attention fusion module fuses the attention features at different scales to determine a fused feature map; and the feature projection head performs global context pooling on the fused feature map, concatenates global average pooling and global max pooling, and obtains the texture feature vector. The texture feature vector is multiplied by the feature vector of each reference image to determine multiple dot product results. These multiple dot product results are then sorted from largest to smallest, and the sorted multiple dot product results are used as the query results for texture-similar images.

[0005] Optionally, the step of fusing attention features of different scales across scales using the cross-scale attention fusion module to determine the fused feature map specifically includes: The attention features of three different scales are resampled by the cross-scale attention fusion module. One set of attention features from the resampled attention features is used as the main feature and the other two sets are used as context features. The context vectors in the context features are extracted by the cross-scale attention fusion module and concatenated. The concatenated context vectors are then concatenated with the main features to determine the fused feature map.

[0006] Optionally, the texture similarity detection model is trained using a reference dataset and a query dataset; wherein the reference dataset includes noisy reference images and viewpoint proximity images, and the query dataset includes a query image; the viewpoint proximity images are images selected from the reference dataset whose viewpoint distance from the query image is less than a preset proximity angle threshold, and the noisy reference images are images selected from the reference dataset whose viewpoint distance from the query image is greater than a preset noise viewpoint threshold; the acquisition of the viewpoint distance specifically includes: ; Where d is the viewing distance. To query the pose corresponding to the image, To query the pose corresponding to the image, tr() is used to find the trace of the matrix, and T is the transpose of the solution matrix.

[0007] Optionally, before training the texture similarity detection model, data augmentation processing is performed on the reference dataset and the query dataset. This data augmentation processing specifically includes: Random scaling and cropping, affine transformation, photometric transformation, information hiding and occlusion simulation.

[0008] Optionally, the parameters of the texture similarity detection model can be adjusted using the Loc loss function; wherein, obtaining the Loc loss function specifically includes: ; in, Let Loc be the loss function, representing the training loss of the texture similarity detection model. To query the feature vector of an image calculated by the model, The feature vectors obtained by the model from images with similar perspectives are represented by these vectors. The total number of reference images used for training in a sample Representing the first of them The feature vector is calculated from a reference image, where τ is the temperature coefficient.

[0009] Optionally, the multi-scale feature extraction module uses a deep convolutional network as the backbone network; wherein the deep convolutional network is a multi-layer CNN such as ResNet18, VGG, or GoogleNet.

[0010] This invention provides a texture similarity detection device, comprising: The acquisition module is used to acquire feature vectors of all offline reference images of the target. A texture feature vector determination module is used to input the query image of the object to be tested into a trained texture similarity detection model for texture similarity detection to obtain the texture feature vector of the reference image. The texture similarity detection model includes a multi-scale feature extraction module, a multi-scale feature processing module, a cross-scale attention fusion module, and a feature projection head connected in sequence. Specifically, the multi-scale feature extraction module extracts features at various scales of the query image to determine a multi-scale feature map; the multi-scale feature processing module processes the multi-scale feature map to determine attention features at different scales; the cross-scale attention fusion module fuses the attention features at different scales to determine a fused feature map; and the feature projection head performs global context pooling on the fused feature map, concatenates global average pooling and global max pooling, and obtains the texture feature vector. The detection query module is used to perform dot product between the texture feature vector and the feature vector of each reference image, determine multiple dot product results, sort the multiple dot product results from largest to smallest, and use the sorted multiple dot product results as the query results of texture similar images.

[0011] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described texture similarity detection method.

[0012] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described texture similarity detection method.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention provides a texture similarity detection method. This method, through the constructed texture similarity detection model, can find reference images that are highly similar to the texture image, significantly improving the efficiency of key point matching, and thus effectively improving the accuracy of texture similarity detection. In addition, the present invention effectively suppresses model overfitting by improving the dataset organization, data augmentation, and training prediction methods, thereby improving the accuracy of texture similarity detection. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0015] Figure 1 A flowchart of a texture similarity detection method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the model structure provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of random sampling based on CAD model and camera pose space provided in an embodiment of the present invention; Figure 4 This is a visualization diagram of training samples provided in an embodiment of the present invention; Figure 5 Training process curves provided for embodiments of the present invention; Figure 6 The training loss variation curves for all sample traversal iterations provided in this embodiment of the invention; Figure 7 The image shows the test results provided in the embodiments of the present invention; Figure 8 This is a schematic diagram illustrating the generalization performance test results of the texture similarity detection model provided in this embodiment of the invention. Figure 9 A schematic diagram of a computer device for a texture similarity detection method provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0017] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart of a texture similarity detection method provided in an embodiment of the present invention, such as... Figure 1 As shown in the figure, this embodiment illustrates a texture similarity detection method, including: S1: Obtain the feature vectors of all offline reference images of the target.

[0019] For example, the texture feature vectors of all reference images for each object in the dataset can be calculated offline and stored locally for easy loading in subsequent steps.

[0020] S2: Input the query image of the object to be tested into the trained texture similarity detection model to obtain the texture feature vector of the reference image.

[0021] Optionally, the texture similarity detection model includes a multi-scale feature extraction module, a multi-scale feature processing module, a cross-scale attention fusion module, and a feature projection head connected in sequence. Specifically, the multi-scale feature extraction module extracts features from the query image at various scales to determine a multi-scale feature map; the multi-scale feature processing module processes the multi-scale feature map to determine attention features at different scales; the cross-scale attention fusion module fuses the attention features at different scales to determine a fused feature map; and the feature projection head performs global context pooling on the fused feature map, concatenates global average pooling and global max pooling, and obtains a texture feature vector.

[0022] For example, such as Figure 2As shown, the multi-scale feature extraction module can use a multi-layer CNN as the backbone network to extract features at various scales of the image. For example, taking ResNet as an example, a pre-trained ResNet18 model can be loaded, its weight parameters fixed, and its key layer {layer2, layer3, layer4} feature maps extracted as {C2, C3, C4}. Alternative solutions for multi-scale feature extraction layers using deep convolutional networks include: VGG, ResNet, GoogleNet, and YOLO series backbone networks, etc.

[0023] Then, through the multi-scale feature processing module, the attention extraction module is applied to the {C2, C3, C4} feature maps to obtain three sets of attention features {A2, A3, A4} at different scales. This module combines channel attention and spatial attention for feature processing and performs residual connections on the input feature maps to further extract rich features while maintaining the diversity of the original feature maps. Then, by employing a cross-scale attention fusion module, cross-scale attention fusion is performed on the above three sets of attention features {A2, A3, A4} at different scales. This module first resamples the attention processing results of each scale feature to ensure that the data dimensions are consistent. It then sequentially selects one set of attention features as the main feature and the other two as context features, extracts the context vectors, concatenates them, and concatenates them with the main feature to generate attention maps {F2, F3, F4}, as shown below. Figure 2 As shown; Finally, global context pooling is performed on {F2, F3, F4} using the feature projection head to obtain one-dimensional feature maps {G2, G3, G4}. The specific process of this pooling operation is to concatenate the results of global average pooling and global max pooling, then concatenate {G2, G3, G4} for feature projection, and finally perform L2 normalization to obtain the final texture feature vector V.

[0024] For example, the texture similarity detection model is trained using a reference dataset and a query dataset. The colmap tool can be used to calculate the pose of captured videos or discrete images to determine the corresponding reference dataset and query dataset. The dataset contains images and corresponding poses. The images are clean foreground images of uniform size with the background removed, and foreground segmentation images containing only a single object. Their corresponding poses are stored separately for later use, and the pose reference of a single object in the reference dataset and the query dataset is consistent.

[0025] For example, during model training, each selected sample consists of a query image, a viewpoint proximity image, and several noisy reference images. All images in this sample are foreground images of the same size belonging to the same object. A proximity angle threshold phi_near and a noise viewpoint threshold phi_noise are set, and the viewpoint distance is obtained using the following formula: Where d is the viewing distance. This represents the pose of the image, i.e., a 3x3 rotation matrix. This represents the pose corresponding to the queried image. The tr() function represents the pose corresponding to the query image, the tr() function calculates the trace of the matrix, and the T function calculates the transpose of the matrix.

[0026] Randomly select a query image, calculate all view distances between all reference images and the query image, and randomly select images with view distances within the range of the proximity angle threshold. All noisy reference images are randomly selected from the set of reference images with view distances greater than the noise view distance threshold. For example, Figure 3 To visualize random sampling in space, a set of reference images is obtained by rendering CAD models under different camera poses for model training and prediction.

[0027] For example, before training the model, data augmentation can be performed to enhance the model's generalization ability, specifically including: (1) random scaling and cropping: the input image is randomly scaled within a specific scaling ratio range, preferably [0.8, 1.2], and then a region with the same size as the original input is randomly cropped from the scaled image. This operation aims to simulate the scale change of objects at different distances and partial occlusion.

[0028] (2) Affine transformation: Apply an affine transformation, including translation and shearing, to the input image. The magnitude of the transformation is preferably controlled between [-0.2, 0.2] of the image size. This operation is used to simulate slight changes in the viewpoint.

[0029] (3) Photometric transformation: This includes brightness, contrast, color, and noise adjustments, specifically including: Brightness adjustment: Randomly adjusting the overall brightness of the image within a specific brightness factor range, preferably [0.7, 1.3]. This operation aims to enhance the model's adaptability to different lighting conditions. Contrast adjustment: Randomly adjusting the contrast of the image within a specific contrast factor range, preferably [0.7, 1.3]. This operation enables the model to handle low-contrast or high-contrast scenes caused by ambient light. Color dithering: Randomly adding small perturbations to the color channels of the image, or randomly adjusting the saturation and hue of the image within the HSV color space. This operation aims to reduce the model's overfitting to specific colors. Adding noise: Adding random Gaussian noise or salt-and-pepper noise to the input image to simulate the noise generated by the image sensor under low lighting conditions, thereby improving the model's anti-interference ability.

[0030] (4) Simulation of information hiding and occlusion: Random occlusion: One or more rectangular occlusion regions are randomly placed on the input image, and the height and width of the occlusion regions are preferably within the range of [0.1, 0.3] of the original image. This operation forces the model to learn multiple components of the target rather than relying on a single local feature for decision-making, greatly enhancing the model's robustness to partial occlusion.

[0031] Random erase: Similar to random occlusion, but replaces the pixel values ​​in the occluded area with random or average values.

[0032] The random combination of the above transformations generates more complex and diverse training samples in each model training iteration to prevent overfitting and enhance the model's generalization performance.

[0033] For example, during training, the number of samples in a batch is determined by the batch_size. All these images are input into the above model processing flow to obtain the texture feature vectors of the query image, the image with a similar viewpoint, and all noisy reference images, calculated as follows: : in, For model training loss, To query the feature vector of an image calculated by the model, The feature vectors obtained by the model from images with similar perspectives are represented by these vectors. This represents the total number of reference images used for training in a given sample. Representing the first of them The feature vector is calculated from a reference image, where τ is the temperature coefficient.

[0034] like Figure 4 As shown, a batch contains 4 samples, each consisting of 1 query image, 1 viewpoint proximity reference image, and 8 noisy reference images; for example... Figure 5 The image shows a visualization of the texture similarity of a set of samples during training. The graph consists of four small correlation scatter plots formed by four samples. The horizontal axis of each small plot represents the viewing angle between all the reference images and the query image involved in the training of a sample, in degrees. The vertical axis is the dot product of the feature vectors of the query image and the reference image, i.e., the similarity measure (dimensionless). The graph shows that the reference image and the query image with similar viewing angles have the highest correlation. As the viewing angle between the reference image and the query image increases, the similarity decreases. This indicates that the model has successfully obtained the correct feature vector representation through training, thus distinguishing the reference image with a similar viewing angle from the noisy reference image with a distant viewing angle.

[0035] Training process curve as follows Figure 6 As shown, a complete epoch is completed after all samples have been traversed. The training process ends when the required number of iterations is reached or the loss remains below a set threshold for a period of time.

[0036] like Figure 7 As shown in (a), the discrimination results are displayed when the object is partially occluded, such as Figure 7 As shown in (b), it demonstrates that the model of the present invention can correctly predict the texture similarity results when the texture of the query image has a certain degree of variation. The raencode model, trained on the publicly available dataset ycbv, is used to infer unseen mouse data, and the matching results are as follows: Figure 8 As shown in (a); when the mouse is partially obscured, the matching result is as follows: Figure 8 As shown in (b).

[0037] S3: Perform dot product between the texture feature vector and the feature vector of each reference image to determine multiple dot product results. Sort the multiple dot product results from largest to smallest and use the sorted multiple dot product results as the query results of texture similar images.

[0038] For example, the target foreground image can be segmented from a certain object first. After removing the background, the effective area of ​​the foreground image is scaled proportionally to the length and width to the size required by the texture similarity detection model. The scaled image is the query image. The query image is input into the trained texture similarity detection model to calculate the texture feature vector of the query image. Then, the dot product of this vector with the feature vectors of all reference images that have been calculated offline for the corresponding target is taken. The dot product results are sorted from largest to smallest to obtain the query results of texture similar images.

[0039] The above describes one or more embodiments of a texture similarity detection method provided in this specification. Based on the same idea, this specification also provides a corresponding texture similarity detection device, including: The acquisition module is used to acquire feature vectors of all offline reference images of the target. A texture feature vector determination module is used to input the query image of the object to be tested into a trained texture similarity detection model for texture similarity detection to obtain the texture feature vector of the reference image. The texture similarity detection model includes a multi-scale feature extraction module, a multi-scale feature processing module, a cross-scale attention fusion module, and a feature projection head connected in sequence. Specifically, the multi-scale feature extraction module extracts features at various scales of the query image to determine a multi-scale feature map; the multi-scale feature processing module processes the multi-scale feature map to determine attention features at different scales; the cross-scale attention fusion module fuses the attention features at different scales to determine a fused feature map; and the feature projection head performs global context pooling on the fused feature map, concatenates global average pooling and global max pooling, and obtains the texture feature vector. The detection query module is used to perform dot product between the texture feature vector and the feature vector of each reference image, determine multiple dot product results, sort the multiple dot product results from largest to smallest, and use the sorted multiple dot product results as the query results of texture similar images.

[0040] Specific limitations regarding the texture similarity detection device can be found in the limitations of the texture similarity detection method described above, and will not be repeated here. Each module in the aforementioned texture similarity detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0041] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the texture similarity detection method provided above.

[0042] The present invention also provides Figure 9 The schematic diagram of the computer device shown is as follows: Figure 9 As shown, at the hardware level, the computer device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the texture similarity detection method provided in the above embodiments.

[0043] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0044] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

Claims

1. A texture similarity detection method, characterized in that, include: Obtain the feature vectors of all offline reference images of the target; The query image of the object to be tested is input into a trained texture similarity detection model to obtain the texture feature vector of the reference image. The texture similarity detection model includes a multi-scale feature extraction module, a multi-scale feature processing module, a cross-scale attention fusion module, and a feature projection head connected in sequence. Specifically, the multi-scale feature extraction module extracts features at various scales of the query image to determine a multi-scale feature map; the multi-scale feature processing module processes the multi-scale feature map to determine attention features at different scales; the cross-scale attention fusion module fuses the attention features at different scales to determine a fused feature map; and the feature projection head performs global context pooling on the fused feature map, concatenates global average pooling and global max pooling, and obtains the texture feature vector. The texture feature vector is multiplied by the feature vector of each reference image to determine multiple dot product results. These multiple dot product results are then sorted from largest to smallest, and the sorted multiple dot product results are used as the query results for texture-similar images.

2. The texture similarity detection method as described in claim 1, characterized in that, The step of fusing attention features at different scales through the cross-scale attention fusion module to determine the fused feature map specifically includes: The attention features of three different scales are resampled by the cross-scale attention fusion module. One set of attention features from the resampled attention features is used as the main feature and the other two sets are used as context features. The context vectors in the context features are extracted by the cross-scale attention fusion module and concatenated. The concatenated context vectors are then concatenated with the main features to determine the fused feature map.

3. The texture similarity detection method as described in claim 1, characterized in that, The texture similarity detection model is trained using a reference dataset and a query dataset. The reference dataset includes noisy reference images and viewpoint proximity images, and the query dataset includes a query image. The viewpoint proximity images are selected from the reference dataset whose viewpoint distance from the query image is less than a preset proximity angle threshold. The noisy reference images are selected from the reference dataset whose viewpoint distance from the query image is greater than a preset noise viewpoint threshold. The acquisition of the viewpoint distance specifically includes: ； Where d is the viewing distance. To query the pose corresponding to the image, To query the pose corresponding to the image, tr() is used to find the trace of the matrix, and T is the transpose of the solution matrix.

4. The texture similarity detection method as described in claim 3, characterized in that, Before training the texture similarity detection model, data augmentation is performed on the reference dataset and the query dataset. The data augmentation specifically includes: Random scaling and cropping, affine transformation, photometric transformation, information hiding and occlusion simulation.

5. The texture similarity detection method as described in claim 1, characterized in that, The parameters of the texture similarity detection model are adjusted using the Loc loss function; specifically, obtaining the Loc loss function includes: ； in, Let Loc be the loss function, representing the training loss of the texture similarity detection model. To query the feature vector of an image calculated by the model, The feature vectors obtained by the model from images with similar perspectives are represented by these vectors. The total number of reference images used for training in a sample Representing the first of them The feature vector is calculated from a reference image, where τ is the temperature coefficient.

6. The texture similarity detection method as described in claim 1, characterized in that, The multi-scale feature extraction module uses a deep convolutional network as its backbone network; wherein, the deep convolutional network is a multi-layer CNN such as ResNet18, VGG, or GoogleNet.

7. A texture similarity detection device, characterized in that, include: The acquisition module is used to acquire feature vectors of all offline reference images of the target. A texture feature vector determination module is used to input the query image of the object to be tested into a trained texture similarity detection model for texture similarity detection to obtain the texture feature vector of the reference image. The texture similarity detection model includes a multi-scale feature extraction module, a multi-scale feature processing module, a cross-scale attention fusion module, and a feature projection head connected in sequence. Specifically, the multi-scale feature extraction module extracts features at various scales of the query image to determine a multi-scale feature map; the multi-scale feature processing module processes the multi-scale feature map to determine attention features at different scales; the cross-scale attention fusion module fuses the attention features at different scales to determine a fused feature map; and the feature projection head performs global context pooling on the fused feature map, concatenates global average pooling and global max pooling, and obtains the texture feature vector. The detection query module is used to perform dot product between the texture feature vector and the feature vector of each reference image, determine multiple dot product results, sort the multiple dot product results from largest to smallest, and use the sorted multiple dot product results as the query results of texture similar images.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the texture similarity detection method according to any one of claims 1-6.

9. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the texture similarity detection method according to any one of claims 1-6.

Images