An intelligent furniture damage detection method

Abstract

The present invention relates to the field of smart homes, and specifically to an intelligent furniture damage detection method. The present invention adopts a multimodal feature fusion method based on dual bidirectional interaction to dynamically weight local features, balance the local features of point clouds, and perform bidirectional interaction of furniture point clouds and RGB images based on two-dimensional and three-dimensional cross-dimensional features. Through bidirectional cross-attention, features of point cloud regions of interest are selectively combined. On the basis of highlighting the color and three-dimensional shape of the furniture, the color and shape information are fully fused and interacted to achieve a more comprehensive feature representation, thereby increasing the accuracy of crack and damage detection. The present invention solves the traditional quantization error problem by combining continuous integerization with sparse pulses, maintains accuracy while improving detection efficiency, and realizes end-to-end optimization from image enhancement to efficient detection. By dynamically fusing multi-scale two-dimensional features, the method adapts to cracks of different forms and improves the detection capability of cracks.

Description

Technical Field

[0001] The present invention belongs to the field of smart homes, and in particular relates to an intelligent furniture damage detection method. Background Art

[0002] Furniture damage detection refers to the use of technical means to detect and identify damage or defects in furniture products to help manufacturers, retailers or consumers discover and deal with problems in a timely manner; existing furniture detection technology has low detection efficiency and detection accuracy needs to be improved. Summary of the Invention

[0003] In response to the above situation, in order to overcome the shortcomings of the prior art, the present invention provides an intelligent furniture damage detection method. To address the problems of low detection efficiency and detection accuracy of existing furniture detection technologies, the present invention creatively adopts a multimodal feature fusion method based on dual bidirectional interaction. Point cloud features are extracted through multiple serial self-attention modules, local features are dynamically weighted, local features of the point cloud are balanced, and two-dimensional-three-dimensional cross-dimensional feature bidirectional interaction is performed on the furniture point cloud and RGB image. Through bidirectional cross-attention, features of the point cloud region of interest are selectively combined. On the basis of highlighting the color and three-dimensional shape of the furniture, the color and shape information are fully fused and interacted. The unique information of each modality is used to achieve a more comprehensive feature representation. The importance of features of different modalities is dynamically weighted to improve the quality of the fused features and increase the accuracy of crack and damage detection. The present invention solves the quantization error problem of traditional SNN by combining continuous integerization with sparse pulses, improves detection efficiency while maintaining accuracy, and achieves end-to-end optimization from image enhancement to efficient detection. At the same time, multi-scale two-dimensional features are dynamically fused to adapt to cracks of different shapes, further improving the detection capability of cracks.

[0004] The present invention provides an intelligent furniture damage detection method, which specifically includes the following steps:

[0005] Step S1: Data acquisition, collecting RGB images of furniture and scanning the furniture to generate point clouds;

[0006] Step S2: Image enhancement, performing brightness enhancement processing on the RGB image based on the color space to obtain a high-resolution RGB image, specifically including the following steps:

[0007] Step S21: color space conversion, converting the RGB image into the YCbCr color space and separating it to obtain the luminance channel and the chrominance channel;

[0008] Step S22: Preprocessing: performing center cropping and bicubic interpolation upsampling on the luminance channel to obtain a luminance tensor;

[0009] Step S23: Brightness enhancement: enhancing the brightness tensor through multi-layer convolution to obtain an enhanced brightness tensor;

[0010] Step S24: Pixel shuffle is used to upsample the enhanced brightness tensor to obtain a high-resolution brightness image.

[0011] Step S25: Chroma fusion, performing bicubic interpolation upsampling on the chroma channel and splicing it with the high-resolution luminance image to obtain a high-resolution RGB image;

[0012] Step S3: Pulse sequence-based multi-scale image feature extraction, performing pulse sequence-based multi-scale image feature extraction on the high-resolution RGB image to obtain multi-scale features, specifically comprising the following steps:

[0013] Step S31: Membrane potential generation, using a spiking neural network to convolve the high-resolution RGB image to generate a continuous membrane potential;

[0014] Step S32: integer processing, quantizing the continuous membrane potential into an integer to obtain an integer continuous membrane potential;

[0015] Step S33: pulse train generation, decomposing the integer continuous membrane potential to obtain a sparse binary pulse train;

[0016] Step S34: local feature extraction, using a convolutional neural network as an encoder to extract features from the sparse binary pulse sequence to obtain a local feature map sequence;

[0017] Step S35: multi-scale fusion, weighted fusion of the local feature map sequence to obtain multi-scale features;

[0018] Step S4: Surface crack detection based on the image, generating an attention score through the attention mechanism, weighting and decoding the multi-scale features according to the attention score, and generating a surface crack detection result;

[0019] Step S5: Multimodal feature fusion based on dual bidirectional interaction is performed on the multi-scale features and the point cloud to obtain three-dimensional point cloud features, which specifically includes the following steps:

[0020] Step S51: point cloud enhanced feature extraction based on self-attention, feature extraction is performed on the point cloud and local feature enhancement based on self-attention is performed to obtain enhanced local features;

[0021] Step S52: Bidirectional interactive fusion of point cloud features, performing bidirectional interactive fusion of point cloud features and enhanced local features based on a cross-attention mechanism to obtain enhanced point cloud features, specifically including the following steps:

[0022] Step S521: Extract cross-attention features guided by point cloud features, construct a query vector for the point cloud features, construct a key vector and a value vector for the enhanced local features, and perform cross-attention processing on the query vector of the point cloud features and the key vector and value vector of the enhanced local features to obtain a global cross-feature.

[0023] Step S522: Extract cross-attention features guided by enhanced local features, repeat step S521, and exchange the enhanced local features in step S521 with the point cloud features to obtain local cross features;

[0024] Step S523: cross-modal connection, performing feature connection operation on the global cross feature and the local cross feature to obtain a cross-modal connection feature;

[0025] Step S524: Cross-modal feature interaction: performing multi-layer perceptron processing on the cross-modal connection features, and performing weighted multiplication with the global cross features and the local cross features, respectively, to obtain global interaction features and local interaction features;

[0026] Step S525: cross-modal fusion, connecting the global interaction features with the local interaction features to obtain cross-modal fusion features, i.e., enhanced point cloud features;

[0027] Step S53: Based on the bidirectional interaction of 2D-3D cross-dimensional features, cross-dimensional feature fusion is performed on the enhanced point cloud features and the multi-scale features to obtain 3D point cloud features, which specifically includes the following steps:

[0028] Step S531: using 1×1 convolution to reduce the dimension of the enhanced point cloud features and adding them to the RGB features to obtain optimized RGB features;

[0029] Step S532: aligning and fusing the RGB features with the depth features, and adding them to the enhanced point cloud features to obtain optimized point cloud features;

[0030] Step S533: cross-dimensional feature extraction, fusing the optimized RGB features with the optimized point cloud features to obtain cross-dimensional optimized features, i.e., three-dimensional point cloud features;

[0031] Step S6: Based on the damage detection of the point cloud, an activation function is used to activate and output the three-dimensional point cloud features to generate a damage detection result.

[0032] The beneficial results achieved by the present invention using the above scheme are as follows:

[0033] (1) The present invention adopts a multimodal feature fusion method based on dual bidirectional interaction. It extracts point cloud features through multiple serial self-attention modules, dynamically weights local features, balances local features of point clouds, and performs bidirectional interaction of two-dimensional and three-dimensional cross-dimensional features on furniture point clouds and RGB images. Through bidirectional cross-attention, it selectively combines features of point cloud regions of interest. On the basis of highlighting the color and three-dimensional shape of furniture, it makes full fusion interaction of color and shape information. It uses the unique information of each modality to achieve more comprehensive feature representation, dynamically weights and learns the importance of features of different modalities, improves the quality of fused features, and increases the accuracy of crack and damage detection.

[0034] (2) The present invention solves the quantization error problem of traditional SNN by combining continuous integerization with sparse pulses, improving detection efficiency while maintaining accuracy, achieving end-to-end optimization from image enhancement to efficient detection, and dynamically fusing multi-scale two-dimensional features to adapt to cracks of different shapes, further improving the detection capability of cracks. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of an intelligent furniture damage detection method provided by the present invention.

[0036] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] Example 1, see Figure 1 The present invention provides an intelligent furniture damage detection method, which specifically includes the following steps:

[0039] Step S1: Collect RGB images of furniture and scan the furniture to generate point clouds;

[0040] Step S2: performing brightness enhancement processing on the RGB image based on the color space to obtain a high-resolution RGB image;

[0041] Step S3: performing multi-scale image feature extraction based on a pulse sequence on the high-resolution RGB image to obtain multi-scale features;

[0042] Step S4: Generate an attention score through the attention mechanism, weight and decode the multi-scale features according to the attention score, and generate the surface crack detection result;

[0043] Step S5: performing multi-modal feature fusion based on dual bidirectional interaction on the multi-scale features and the point cloud to obtain three-dimensional point cloud features;

[0044] Step S6: Use an activation function to activate and output the three-dimensional point cloud features to generate damage detection results.

[0045] Embodiment 2: This embodiment is based on the above embodiment, and step S2 specifically includes the following steps:

[0046] Step S21: Convert the RGB image into the YCbCr color space and separate it to obtain the luminance channel and the chrominance channel;

[0047] Step S22: performing center cropping and bicubic interpolation upsampling on the luminance channel to obtain a luminance tensor;

[0048] Step S23: enhancing the brightness tensor through multi-layer convolution to obtain an enhanced brightness tensor;

[0049] Step S24: Use Pixel Shuffle to upsample the enhanced brightness tensor to obtain a high-resolution brightness image;

[0050] Step S25: Perform bicubic interpolation upsampling on the chroma channel and concatenate it with the high-resolution luminance image to obtain a high-resolution RGB image.

[0051] Embodiment 3: This embodiment is based on the above embodiment, and step S3 specifically includes the following steps:

[0052] Step S31: convolve the high-resolution RGB image using a spiking neural network to generate a continuous membrane potential;

[0053] Step S32: quantizing the continuous membrane potential into an integer to obtain an integerized continuous membrane potential;

[0054] Step S33: decomposing the integer continuous membrane potential to obtain a sparse binary pulse train;

[0055] Step S34: using a convolutional neural network as an encoder to extract features from the sparse binary pulse sequence to obtain a local feature map sequence;

[0056] Step S35: Perform weighted fusion on the local feature map sequence to obtain multi-scale features.

[0057] Example 4: This example is based on the above example. Step S5: Multimodal feature fusion based on dual bidirectional interaction is performed on the multi-scale features and the point cloud to obtain a three-dimensional point cloud feature. Specifically, the following steps are included:

[0058] Step S51: point cloud enhanced feature extraction based on self-attention, feature extraction is performed on the point cloud and local feature enhancement based on self-attention is performed to obtain enhanced local features;

[0059] Step S52: Bidirectional interactive fusion of point cloud features, performing bidirectional interactive fusion of point cloud features and enhanced local features based on a cross-attention mechanism to obtain enhanced point cloud features, specifically including the following steps:

[0060] Step S521: Extract cross-attention features guided by point cloud features, construct a query vector for the point cloud features, construct a key vector and a value vector for the enhanced local features, and perform cross-attention processing on the query vector of the point cloud features and the key vector and value vector of the enhanced local features to obtain a global cross-feature.

[0061] Step S522: Extract cross-attention features guided by enhanced local features, repeat step S521, and exchange the enhanced local features in step S521 with the point cloud features to obtain local cross features;

[0062] Step S523: cross-modal connection, performing feature connection operation on the global cross feature and the local cross feature to obtain a cross-modal connection feature;

[0063] Step S524: Cross-modal feature interaction: performing multi-layer perceptron processing on the cross-modal connection features, and performing weighted multiplication with the global cross features and the local cross features, respectively, to obtain global interaction features and local interaction features;

[0064] Step S525: cross-modal fusion, connecting the global interaction features with the local interaction features to obtain cross-modal fusion features, i.e., enhanced point cloud features;

[0065] Step S53: Based on the bidirectional interaction of 2D-3D cross-dimensional features, cross-dimensional feature fusion is performed on the enhanced point cloud features and the multi-scale features to obtain 3D point cloud features, which specifically includes the following steps:

[0066] Step S531: using 1×1 convolution to reduce the dimension of the enhanced point cloud features and adding them to the RGB features to obtain optimized RGB features;

[0067] Step S532: aligning and fusing the RGB features with the depth features, and adding them to the enhanced point cloud features to obtain optimized point cloud features;

[0068] Step S533: cross-dimensional feature extraction, fusing the optimized RGB features with the optimized point cloud features to obtain cross-dimensional optimized features, i.e., three-dimensional point cloud features.

[0069] Example 5: Based on the above example, this example uses the present invention to detect surface cracks on wooden chairs:

[0070] Scenario description: Check whether there are surface cracks on the seat and backrest of an old wooden chair;

[0071] Implementation steps:

[0072] Data acquisition: Use an RGB camera to capture images of the chair from multiple angles, and use a 3D scanner to generate point cloud data of the chair;

[0073] Image enhancement: Brightness enhancement is performed on the captured RGB image (step S2) to improve the contrast of the crack area;

[0074] Crack detection: Multi-scale features are extracted through pulse sequences (step S3) and combined with an attention mechanism (step S4) to locate fine cracks on the edge of the seat surface;

[0075] Multimodal fusion: Fusion of image features with point cloud data (step S5) to confirm whether the cracks penetrate into the wood;

[0076] Result output: The system marks the crack location and assesses the extent of damage, prompting "There is a 3cm long surface crack on the right side of the seat surface, which needs to be repaired."

[0077] Example 6: Based on the above example, this example uses the present invention to detect edge chipping of a glass coffee table:

[0078] Scenario description: Detecting chips or scratches on the edge of a tempered glass coffee table;

[0079] Implementation steps:

[0080] Data collection: Use a high-resolution camera to capture reflected light images of the glass edge, and combine it with laser scanning to generate a high-precision point cloud;

[0081] Multi-scale analysis: Extracting edge microtexture features through pulse sequences (step S3) to enhance the visibility of small chipping;

[0082] 3D verification: Fusion of suspected chipped areas in the image with point cloud curvature change features (step S533) to eliminate false detections caused by reflections;

[0083] Result visualization: Output the chipping position and depth marked in the 3D model, and prompt "The chipping depth of the lower right corner edge is 2mm, and polishing is required."

[0084] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0085] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0086] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. An intelligent furniture damage detection method, characterized by: The specific steps include: Step S1: Collect RGB images of furniture and scan the furniture to generate a point cloud; Step S2: performing brightness enhancement processing on the RGB image based on the color space to obtain a high-resolution RGB image; Step S3: performing multi-scale image feature extraction based on a pulse sequence on the high-resolution RGB image to obtain multi-scale features, specifically comprising the following steps: Step S31: convolve the high-resolution RGB image using a spiking neural network to generate a continuous membrane potential; Step S32: quantizing the continuous membrane potential into an integer to obtain an integerized continuous membrane potential; Step S33: decomposing the integer continuous membrane potential to obtain a sparse binary pulse train; Step S34: using a convolutional neural network as an encoder to extract features from the sparse binary pulse sequence to obtain a local feature map sequence; Step S35: performing weighted fusion on the local feature map sequence to obtain multi-scale features; Step S4: Generate an attention score through the attention mechanism, weight and decode the multi-scale features according to the attention score, and generate the surface crack detection result; Step S5: Perform multi-modal feature fusion based on dual bidirectional interaction on the multi-scale features and the point cloud to obtain three-dimensional point cloud features, which specifically includes the following steps: Step S51: extracting features from the point cloud and performing local feature enhancement based on self-attention to obtain enhanced local features; Step S52: performing a bidirectional interactive fusion of the point cloud features and the enhanced local features based on a cross-attention mechanism to obtain enhanced point cloud features, specifically comprising the following steps: Step S521: constructing a query vector for the point cloud feature, constructing a key vector and a value vector for the enhanced local feature, and performing a cross-attention mechanism on the query vector of the point cloud feature and the key vector and value vector of the enhanced local feature to obtain a global cross feature; Step S522: repeating step S521, and exchanging the enhanced local features in step S521 with the point cloud features to obtain local cross features; Step S523: performing a feature connection operation on the global cross feature and the local cross feature to obtain a cross-modal connection feature; Step S524: performing multi-layer perceptron processing on the cross-modal connection features, and performing weighted multiplication with the global cross features and the local cross features, respectively, to obtain global interaction features and local interaction features; Step S525: Connecting the global interaction features with the local interaction features to obtain cross-modal fusion features, i.e., enhanced point cloud features; Step S53: performing cross-dimensional feature fusion on the enhanced point cloud features and the multi-scale features to obtain three-dimensional point cloud features; Step S6: Use an activation function to activate and output the three-dimensional point cloud features to generate damage detection results.

2. The intelligent furniture damage detection method according to claim 1, characterized in that: Step S2 specifically includes the following steps: Step S21: Convert the RGB image into the YCbCr color space and separate it to obtain the luminance channel and the chrominance channel; Step S22: performing center cropping and bicubic interpolation upsampling on the luminance channel to obtain a luminance tensor; Step S23: enhancing the brightness tensor through multi-layer convolution to obtain an enhanced brightness tensor; Step S24: Use Pixel Shuffle to upsample the enhanced brightness tensor to obtain a high-resolution brightness image; Step S25: Perform bicubic interpolation upsampling on the chroma channel and concatenate it with the high-resolution luminance image to obtain a high-resolution RGB image.

Images