Intelligent furniture damage detection method

Abstract

The invention relates to the field of smart home, in particular to an intelligent furniture damage detection method, which adopts a multi-modal feature fusion method based on double bidirectional interaction, dynamically weights local features, balances the local features of point clouds, and performs two-dimensional-three-dimensional cross-dimensional feature bidirectional interaction on furniture point clouds and RGB (red, green and blue) images, so that the furniture damage detection accuracy is improved. Through bidirectional cross attention and selective combination of interested point cloud region features, on the basis of highlighting the color and three-dimensional shape of furniture, color and shape information is fully fused and interacted, more comprehensive feature representation is realized, and the accuracy of crack and damage detection is improved; according to the method, through combination of continuous integers and sparse pulses, the traditional quantization error problem is solved, the precision is maintained while the detection efficiency is improved, end-to-end optimization from image enhancement to efficient detection is realized, and through dynamic fusion of multi-scale two-dimensional features, the method adapts to cracks in different forms, and the crack detection capability is improved.

Description

Technical Field

[0001] The present invention belongs to the field of smart home, and specifically relates to an intelligent furniture damage detection method. Background Art

[0002] Furniture damage detection refers to using technical means to detect and identify damages or defects existing in furniture products to help manufacturers, retailers or consumers discover and handle problems in a timely manner; the existing furniture detection technologies have low detection efficiency and the detection accuracy needs to be improved. Summary of the Invention

[0003] In view of the above situation, to overcome the defects of the prior art, the present invention provides an intelligent furniture damage detection method. Aiming at the problems of low detection efficiency and the need to improve detection accuracy of the existing furniture detection technologies, the present invention creatively adopts a multi-modal feature fusion method based on dual bidirectional interaction, extracts point cloud features through multiple cascaded self-attention modules, dynamically weights local features, balances the local features of the point cloud, and performs two-dimensional to three-dimensional cross-dimensional feature bidirectional interaction on furniture point clouds and RGB images. Through bidirectional cross-attention, selectively combines the features of the point cloud regions of interest, on the basis of highlighting the color and three-dimensional shape of the furniture, enables the information of color and shape to be fully fused and interacted, uses the unique information of each modality to achieve a more comprehensive feature representation, dynamically weights and learns the feature importance of different modalities, improves the quality of the fused features, and increases the accuracy of crack and damage detection; the present invention combines continuous integerization and sparse pulses to solve the quantization error problem of traditional SNNs, maintains the accuracy while improving the detection efficiency, realizes end-to-end optimization from image enhancement to efficient detection, and at the same time dynamically fuses multi-scale two-dimensional features to adapt to cracks of different shapes, further improving the detection ability for cracks.

[0004] An intelligent furniture damage detection method provided by the present invention specifically includes the following steps:

[0005] Step S1: Data acquisition, collect RGB images of furniture and scan the furniture to generate point clouds;

[0006] Step S2: Image enhancement, perform brightness enhancement processing on the RGB image based on the color space to obtain a high-resolution RGB image, which specifically includes the following steps:

[0007] Step S21: Color space conversion, convert the RGB image to the YCbCr color space and separate it to obtain a luminance channel and a chrominance channel;

[0008] Step S22: Preprocessing, perform central cropping and bicubic interpolation upsampling on the luminance channel to obtain a luminance tensor;

[0009] Step S23: Brightness enhancement. The brightness tensor is enhanced through multi-layer convolution to obtain an enhanced brightness tensor.

[0010] Step S24: Pixel rearrangement. Pixel Shuffle is used to upsample the enhanced brightness tensor to obtain a high-resolution brightness image.

[0011] Step S25: Chromaticity fusion. The chromaticity channel is upsampled by bicubic interpolation and stitched with the high-resolution brightness image to obtain a high-resolution RGB image.

[0012] Step S3: Multi-scale image feature extraction based on pulse sequences. Multi-scale features are obtained by performing multi-scale image feature extraction based on pulse sequences on the high-resolution RGB image. The specific steps are as follows:

[0013] Step S31: Membrane potential generation. The high-resolution RGB image is convolved using a spiking neural network to generate a continuous membrane potential.

[0014] Step S32: Integerization processing. The continuous membrane potential is quantized to an integer to obtain an integerized continuous membrane potential.

[0015] Step S33: Pulse sequence generation. The integerized continuous membrane potential is decomposed to obtain a sparse binary pulse sequence.

[0016] Step S34: Local feature extraction. A convolutional neural network is used as an encoder to extract features from the sparse binary pulse sequence to obtain a sequence of local feature maps.

[0017] Step S35: Multi-scale fusion. The sequence of local feature maps is weighted and fused to obtain multi-scale features.

[0018] Step S4: Surface crack detection based on images. Attention scores are calculated and generated through an attention mechanism. The multi-scale features are weighted according to the attention scores and decoded to generate surface crack detection results.

[0019] Step S5: Multi-modal feature fusion based on dual bidirectional interaction. Multi-modal feature fusion based on dual bidirectional interaction is performed on the multi-scale features and the point cloud to obtain three-dimensional point cloud features. The specific steps are as follows:

[0020] Step S51: Point cloud enhanced feature extraction based on self-attention. Features of the point cloud are extracted and local features are enhanced based on self-attention to obtain enhanced local features.

[0021] Step S52: Bidirectional interaction fusion of point cloud features. Bidirectional interaction fusion based on a cross-attention mechanism is performed on the point cloud features and the enhanced local features to obtain enhanced point cloud features. The specific steps are as follows:

[0022] Step S521: Cross-attention feature extraction guided by point cloud features. Construct a query vector for the point cloud features, construct key and value vectors for the enhanced local features, and perform cross-attention mechanism processing on the query vector of the point cloud features and the key and value vectors of the enhanced local features to obtain global cross features;

[0023] Step S522: Cross-attention feature extraction guided by enhanced local features. Repeat Step S521 and swap the enhanced local features and point cloud features in Step S521 to obtain local cross features;

[0024] Step S523: Cross-modal connection. Perform a feature connection operation on the global cross features and local cross features to obtain cross-modal connection features;

[0025] Step S524: Cross-modal feature interaction. Perform multi-layer perceptron processing on the cross-modal connection features and multiply them weighted with the global cross features and local cross features respectively to obtain global interaction features and local interaction features;

[0026] Step S525: Cross-modal fusion. Connect the global interaction features and local interaction features to obtain cross-modal fusion features, that is, enhanced point cloud features;

[0027] Step S53: Cross-dimensional feature two-way interaction based on 2D-3D. Perform cross-dimensional feature fusion on the enhanced point cloud features and multi-scale features to obtain 3D point cloud features, which specifically include the following steps:

[0028] Step S531: Use 1×1 convolution to reduce the dimension of the enhanced point cloud features and add them to the RGB features to obtain optimized RGB features;

[0029] Step S532: Register and fuse the RGB features and depth features, and add them to the enhanced point cloud features to obtain optimized point cloud features;

[0030] Step S533: Cross-dimensional feature extraction. Fuse the optimized RGB features and optimized point cloud features to obtain cross-dimensional optimized features, that is, 3D point cloud features;

[0031] Step S6: Damage detection based on point cloud. Use an activation function to activate and output the 3D point cloud features to generate damage detection results.

[0032] The beneficial effects achieved by the present invention using the above solution are as follows:

[0033] (1) The present invention adopts a multi-modal feature fusion method based on dual bidirectional interaction. It extracts point cloud features through multiple cascaded self-attention modules, dynamically weights local features, balances the local features of the point cloud, and performs two-dimensional to three-dimensional cross-dimensional feature bidirectional interaction on furniture point clouds and RGB images. Through bidirectional cross-attention, it selectively combines the features of the point cloud regions of interest, fully fuses and interacts the color and shape information on the basis of highlighting the color and three-dimensional shape of the furniture, uses the unique information of each modality to achieve a more comprehensive feature representation, dynamically weights and learns the importance of features of different modalities, improves the quality of the fused features, and increases the accuracy of crack and damage detection.

[0034] (2) The present invention combines continuous integerization and sparse pulses to solve the quantization error problem of traditional SNNs, maintains the accuracy while improving the detection efficiency, realizes end-to-end optimization from image enhancement to efficient detection, and dynamically fuses multi-scale two-dimensional features to adapt to cracks of different shapes, further improving the detection ability of cracks. Brief Description of the Drawings

[0035] Figure 1 It is a flowchart of an intelligent furniture damage detection method provided by the present invention.

[0036] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention, and do not constitute a limitation to the present invention. Detailed Embodiments

[0037] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments; based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0038] Embodiment 1, refer to Figure 1 , an intelligent furniture damage detection method provided by the present invention specifically includes the following steps:

[0039] Step S1: Collect the RGB image of the furniture and scan the furniture to generate a point cloud.

[0040] Step S2: Perform brightness enhancement processing on the RGB image based on the color space to obtain a high-resolution RGB image.

[0041] Step S3: Perform multi-scale image feature extraction on the high-resolution RGB image based on the pulse sequence to obtain multi-scale features.

[0042] Step S4: Calculate and generate attention scores through the attention mechanism, weight the multi-scale features according to the attention scores and perform decoding to generate surface crack detection results;

[0043] Step S5: Perform multi-modal feature fusion of the multi-scale features and the point cloud based on double bidirectional interaction to obtain three-dimensional point cloud features;

[0044] Step S6: Use an activation function to activate and output the three-dimensional point cloud features to generate damage detection results.

[0045] Embodiment 2. This embodiment is based on the above embodiment. Step S2 specifically includes the following steps:

[0046] Step S21: Convert the RGB image to the YCbCr color space and separate it to obtain a luminance channel and a chrominance channel;

[0047] Step S22: Perform central cropping and bicubic interpolation upsampling on the luminance channel to obtain a luminance tensor;

[0048] Step S23: Enhance the luminance tensor through multiple layers of convolution to obtain an enhanced luminance tensor;

[0049] Step S24: Use Pixel Shuffle to upsample the enhanced luminance tensor to obtain a high-resolution luminance image;

[0050] Step S25: Perform bicubic interpolation upsampling on the chrominance channel and splice it with the high-resolution luminance image to obtain a high-resolution RGB image.

[0051] Embodiment 3. This embodiment is based on the above embodiment. Step S3 specifically includes the following steps:

[0052] Step S31: Use a spiking neural network to perform convolution on the high-resolution RGB image to generate a continuous membrane potential;

[0053] Step S32: Quantize the continuous membrane potential into an integer to obtain an integerized continuous membrane potential;

[0054] Step S33: Decompose the integerized continuous membrane potential to obtain a sparse binary pulse sequence;

[0055] Step S34: Use a convolutional neural network as an encoder to extract features from the sparse binary pulse sequence to obtain a sequence of local feature maps;

[0056] Step S35: Perform weighted fusion on the sequence of local feature maps to obtain multi-scale features.

[0057] Embodiment 4. This embodiment is based on the above embodiments. Step S5: Based on dual bidirectional interaction for multi-modal feature fusion, perform multi-modal feature fusion on the multi-scale features and the point cloud based on dual bidirectional interaction to obtain 3D point cloud features, which specifically includes the following steps:

[0058] Step S51: Extract point cloud enhanced features based on self-attention, extract features from the point cloud and perform local feature enhancement based on self-attention to obtain enhanced local features;

[0059] Step S52: Bidirectional interaction fusion of point cloud features. Perform bidirectional interaction fusion based on the cross-attention mechanism on the point cloud features and the enhanced local features to obtain enhanced point cloud features, which specifically includes the following steps:

[0060] Step S521: Extract cross-attention features guided by point cloud features. Construct a query vector for the point cloud features, construct key vectors and value vectors for the enhanced local features, and perform cross-attention mechanism processing on the query vector of the point cloud features and the key vectors and value vectors of the enhanced local features to obtain global cross features;

[0061] Step S522: Extract cross-attention features guided by enhanced local features. Repeat Step S521 and exchange the enhanced local features and the point cloud features in Step S521 to obtain local cross features;

[0062] Step S523: Cross-modal connection. Perform feature connection operations on the global cross features and the local cross features to obtain cross-modal connection features;

[0063] Step S524: Cross-modal feature interaction. Perform multi-layer perceptron processing on the cross-modal connection features and perform weighted multiplication with the global cross features and the local cross features respectively to obtain global interaction features and local interaction features;

[0064] Step S525: Cross-modal fusion. Connect the global interaction features and the local interaction features to obtain cross-modal fusion features, that is, enhanced point cloud features;

[0065] Step S53: Cross-dimensional feature bidirectional interaction based on 2D-3D. Perform cross-dimensional feature fusion on the enhanced point cloud features and the multi-scale features to obtain 3D point cloud features, which specifically includes the following steps:

[0066] Step S531: Use 1×1 convolution to reduce the dimension of the enhanced point cloud features and add them to the RGB features to obtain optimized RGB features;

[0067] Step S532: Register and fuse the RGB features and the depth features, and add them to the enhanced point cloud features to obtain optimized point cloud features;

[0068] Step S533: Cross - dimensional feature extraction, fusing the optimized RGB features and the optimized point cloud features to obtain cross - dimensional optimized features, that is, three - dimensional point cloud features.

[0069] Example Five: Based on the above example, the present invention is used to detect surface cracks on a wooden chair:

[0070] Scene description: Detect whether there are surface cracks on the seat surface and backrest of an old wooden chair;

[0071] Implementation steps:

[0072] Data collection: Use an RGB camera to take multi - angle images of the chair, and at the same time generate point cloud data of the chair through a 3D scanner;

[0073] Image enhancement: Perform brightness enhancement on the captured RGB images (Step S2) to improve the contrast of the crack area;

[0074] Crack detection: Extract multi - scale features through a pulse sequence (Step S3), and combine with an attention mechanism (Step S4) to locate the fine cracks on the edge of the seat surface;

[0075] Multi - modal fusion: Fuse the image features and the point cloud data (Step S5) to confirm whether the crack penetrates into the interior of the wood;

[0076] Result output: The system marks the crack position and evaluates the damage degree, and prompts "There is a surface crack 3 cm long on the right side of the seat surface, which needs to be repaired".

[0077] Example Six: Based on the above example, the present invention is used to detect chipping on the edge of a glass coffee table:

[0078] Scene description: Detect chipping or scratches on the edge of a tempered glass coffee table;

[0079] Implementation steps:

[0080] Data collection: Use a high - resolution camera to capture the reflected light image of the glass edge, and combine with laser scanning to generate high - precision point cloud;

[0081] Multi - scale analysis: Extract the microscopic texture features of the edge through a pulse sequence (Step S3) to enhance the visibility of small chipping;

[0082] Three - dimensional verification: Fuse the suspected chipping area in the image with the point cloud curvature change features (Step S533) to exclude false detections caused by reflection;

[0083] Result visualization: Output the marked chipping position and depth in the three - dimensional model, and prompt "The chipping depth at the lower - right corner edge is 2 mm, which needs to be polished".

[0084] It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device.

[0085] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

[0086] The above description of the present invention and its embodiments is not restrictive. What is shown in the drawings is only one of the embodiments of the present invention, and the actual structure is not limited thereto. Generally speaking, if those of ordinary skill in the art are inspired by it and, without departing from the purpose of the present invention, design similar structural modes and embodiments to this technical solution without creative efforts, they shall fall within the protection scope of the present invention.

Claims

1. An intelligent furniture damage detection method, characterized in that: Specifically, it includes the following steps: Step S1: Collect the RGB image of the furniture and generate a point cloud by scanning the furniture; Step S2: Perform brightness enhancement processing on the RGB image based on the color space to obtain a high-resolution RGB image; Step S3: Perform multi-scale image feature extraction on the high-resolution RGB image based on the pulse sequence to obtain multi-scale features; Step S4: Calculate and generate an attention score through the attention mechanism, weight the multi-scale features according to the attention score and perform decoding to generate a surface crack detection result; Step S5: Perform multi-modal feature fusion on the multi-scale features and the point cloud based on double bidirectional interaction to obtain three-dimensional point cloud features; Step S6: Use an activation function to activate and output the three-dimensional point cloud features to generate a damage detection result.

2. The intelligent furniture damage detection method according to claim 1, characterized in that: Step S2 specifically includes the following steps: Step S21: Convert the RGB image to the YCbCr color space and separate it to obtain a luminance channel and a chrominance channel; Step S22: Perform central cropping and bicubic interpolation upsampling on the luminance channel to obtain a luminance tensor; Step S23: Enhance the luminance tensor through multiple layers of convolution to obtain an enhanced luminance tensor; Step S24: Use Pixel Shuffle to upsample the enhanced luminance tensor to obtain a high-resolution luminance image; Step S25: Perform bicubic interpolation upsampling on the chrominance channel and splice it with the high-resolution luminance image to obtain a high-resolution RGB image.

3. The intelligent furniture damage detection method according to claim 2, wherein: Step S3 specifically includes the following steps: Step S31: Use a spiking neural network to perform convolution on the high-resolution RGB image to generate a continuous membrane potential; Step S32: Quantize the continuous membrane potential into an integer to obtain an integerized continuous membrane potential; Step S33: Decompose the integerized continuous membrane potential to obtain a sparse binary pulse sequence; Step S34: Use a convolutional neural network as an encoder to perform feature extraction on the sparse binary pulse sequence to obtain a sequence of local feature maps; Step S35: Perform weighted fusion on the sequence of local feature maps to obtain multi-scale features.

4. An intelligent furniture damage detection method according to claim 3, characterized in that: Step S5 specifically includes the following steps: Step S51: Perform feature extraction on the point cloud and perform local feature enhancement based on self-attention to obtain enhanced local features; Step S52: Perform bidirectional interactive fusion on the point cloud features and the enhanced local features based on the cross-attention mechanism to obtain enhanced point cloud features; Step S53: Perform cross-dimensional feature fusion on the enhanced point cloud features and the multi-scale features to obtain three-dimensional point cloud features.

5. The intelligent furniture damage detection method according to claim 4, characterized in that: Step S52 specifically includes the following steps: Step S521: Construct a query vector for the point cloud features, construct key vectors and value vectors for the enhanced local features, and perform cross-attention mechanism processing on the query vector of the point cloud features and the key vectors and value vectors of the enhanced local features to obtain global cross features; Step S522: Repeat Step S521 and exchange the enhanced local features and the point cloud features in Step S521 to obtain local cross features; Step S523: Perform a feature connection operation on the global cross features and the local cross features to obtain cross-modal connection features; Step S524: Perform multi-layer perceptron processing on the cross-modal connection features, and perform weighted multiplication with the global cross features and the local cross features respectively to obtain the global interaction feature and the local interaction feature; Step S525: Connect the global interaction feature and the local interaction feature to obtain the cross-modal fusion feature, that is, the enhanced point cloud feature.

Images