Sketch recognition method and device based on fusion of three-dimensional sparse convolution and two-dimensional convolution

Abstract

The application discloses a sketch recognition method and device based on three-dimensional sparse convolution and two-dimensional convolution fusion, obtains a sketch image, extracts features of the sketch image to obtain sketch image features, simultaneously extracts sketch point features and voxelizes, extracts geometric information of each point on a stroke of the sketch, voxelizes the sketch points based on the geometric information to obtain point voxels, then extracts features of voxels of each point of the sketch to obtain sketch voxel features, finally, fuses the sketch image features and the sketch voxel features, and classifies and recognizes the sketch through a classifier. The application fully utilizes features of the sketch and all sparse points of the sketch, removes large blank areas of the sketch through Sketch-SparseVoxelNet, directly extracts spatial relationships of all points of the sketch, and fuses texture and color features extracted by a convolutional neural network, so that high-precision intelligent recognition of hand-drawn sketches is realized, and designers are better assisted.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology based on neural networks, specifically relating to a sketch recognition method and apparatus based on the fusion of three-dimensional sparse convolution and two-dimensional convolution. Background Technology

[0002] Hand-drawn sketches have become a common tool in multimedia applications and industrial design, especially with the widespread use of touchpad devices such as smartphones and tablets. Hand-drawn sketches can bridge language barriers, helping designers better realize their ideas and create innovative designs. Sketch recognition is an important area of ​​research in sketching, aiming to identify the target that the designer wants to draw, thus better assisting the designer in the design process.

[0003] However, hand-drawn sketches often result in a lot of blank space and the distribution of sketch points is too sparse, which poses a huge challenge to the accurate recognition of sketches. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies and achieve the goal of accurately identifying the content that designers want to draw in hand-drawn sketches of different styles and containing a large amount of blank space, this invention adopts the following technical solution:

[0005] A sketch recognition method based on the fusion of 3D sparse convolution and 2D convolution includes the following steps:

[0006] Step 1: Obtain the hand-drawn sketch image;

[0007] Step 2: Extract features from the hand-drawn sketch image to obtain the sketch image features;

[0008] Step 3: Sketch point feature extraction and voxelization; Extract the geometric information of each point on the stroke of the sketch, and voxelize the sketch points based on the geometric information to obtain the voxels of the points;

[0009] Step 4: Extract features from voxels at each point of the sketch to obtain sketch voxel features;

[0010] Step 5: Sketch classification; fuse sketch image features and sketch voxel features, and use a classifier to classify and recognize sketches.

[0011] Furthermore, in step 3, threshold segmentation is used to extract points on the sketch strokes, and the grayscale value and position information of each point on the sketch image are obtained. The centroid of the sketch is calculated using the position information. The grayscale value, position information, distance between each point and the centroid, and position gradient of each point are used as the geometric information of each point, and the geometric information is voxelized. The distance between each point and the centroid can characterize the overall compactness of the sketch, and the position gradient information of each point can characterize whether each point in the sketch is on the stroke edge. The features formed by the geometric information of each point are converted into voxel features, which helps Sketch-SparseVoxelNet to better extract the spatial features of the sketch and achieve more accurate sketch recognition.

[0012] Furthermore, the position information is the coordinates in the x-axis and y-axis directions of the sketch plane, and the centroid is the average value of the coordinates of all points.

[0013] Furthermore, based on the voxels corresponding to the stroke points of the sketch segmentation, a first hash table is constructed. The three-dimensional sparse convolutional neural network extracts the voxel features of the current stroke point and the voxel features of the current stroke point and other stroke points through the first hash table. A second hash table is constructed on the voxel features. Based on the second hash table, the spatial relationship between the voxel features corresponding to each stroke point is constructed, and finally the sketch voxel features are obtained.

[0014] Furthermore, in step 4, a voxel feature extraction network is constructed based on a set of three-dimensional sparse convolutional neural networks to extract features from the converted voxels and perform voxel downsampling step by step. The extracted features are averaged to obtain the voxel feature vector of the sparse sketch. A set of three-dimensional sparse convolutional neural networks can improve the sketch recognition accuracy by continuously stacking modules to better extract sketch features.

[0015] Furthermore, the aforementioned 3D sparse convolutional neural network extracts features only from the positions of sketch strokes, thereby directly ignoring large blank areas of the sketch, improving efficiency and increasing accuracy. The formula is as follows:

[0016]

[0017] Among them, w ijz The values ​​represent the weights of the convolution kernel at the three axes, k represents the kernel size, and x represents the weights of the convolution kernel at the three axes. ijz The eigenvalues ​​represent the voxel feature maps.

[0018] Furthermore, the three-dimensional sparse convolutional neural network comprises a basic three-dimensional sparse convolution module (BS3DConv), three-dimensional sparse convolution, and batch regularization stacked together. This effectively utilizes the three-dimensional sparse convolution to extract spatial relationships of the sketch, while also introducing residual connections by drawing on the characteristics of the two-dimensional residual module.

[0019] Furthermore, in step 2, sketch image features are extracted using a basic convolutional neural network, and image downsampling is performed step by step to obtain a high-dimensional feature map of the sketch. Global average pooling is then performed on the high-dimensional feature map to obtain the sketch feature vector.

[0020] Furthermore, in step 5, a loss is constructed based on sketch image features, sketch voxel features, and their fused features to train a classification and recognition model, and the trained classification and recognition model is used to classify and recognize the sketch image.

[0021] A sketch recognition device based on the fusion of three-dimensional sparse convolution and two-dimensional convolution includes a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they implement the sketch recognition method based on the fusion of three-dimensional sparse convolution and two-dimensional convolution.

[0022] The advantages and beneficial effects of this invention are as follows:

[0023] The sketch recognition method and apparatus based on the fusion of three-dimensional sparse convolution and two-dimensional convolution of the present invention make full use of the features of the sketch and all the sparse points of the sketch. By removing large blank areas of the sketch through Sketch-SparseVoxelNet, the spatial relationship of all points of the sketch is directly extracted, and the texture and color features extracted by the convolutional neural network are fused to achieve high-precision intelligent recognition of hand-drawn sketches, so as to better assist designers. Attached Figure Description

[0024] Figure 1 This is a Sketch-SparseNet graph, a sketch recognition framework based on the fusion of three-dimensional sparse convolution and two-dimensional convolution, as described in this embodiment of the invention.

[0025] Figure 2 This is a schematic diagram of sketch geometry information in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of point cloud voxelization in an embodiment of the present invention.

[0027] Figure 4 This is a schematic diagram of the Sketch-SparseVoxelNet sketch voxel feature branches in an embodiment of the present invention.

[0028] Figure 5 This is a schematic diagram of sparse voxel 3D convolution in an embodiment of the present invention.

[0029] Figure 6a This is a schematic diagram of the structure of the three-dimensional sparse convolution module in an embodiment of the present invention.

[0030] Figure 6b This is a schematic diagram of the basic three-dimensional sparse convolution module structure in an embodiment of the present invention.

[0031] Figure 7 This is a schematic diagram of the device in an embodiment of the present invention. Detailed Implementation

[0032] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0033] like Figure 1 As shown, a sketch recognition method based on the fusion of 3D sparse convolution and 2D convolution is proposed. This method utilizes Sketch-SparseNet, a voxel-based sketch recognition framework that fuses 2D and 3D sparse convolutions, to perform sketch recognition. The specific steps include:

[0034] Step 1: Acquiring the hand-drawn sketch image. Acquire the hand-drawn sketch using a touchscreen device such as a graphics tablet and save it as an image.

[0035] In this embodiment of the invention, after the drawing tablet collects the hand-drawn sketch, it saves the sketch canvas as a sketch image, resulting in an image of size 256*256.

[0036] Step 2: Convolutional Neural Network Feature Extraction. A convolutional neural network (CNN) is used to extract features from the hand-drawn sketch image, resulting in a feature vector of the sketch.

[0037] In this embodiment of the invention, a convolutional neural network (CNN) is used to extract features from the sketch image using basic convolutional modules, and downsampling is performed step by step to obtain a high-dimensional feature map of the sketch. The 256*256 feature map is downsampled step by step into an 8*8 feature map, and global average pooling is performed on the 512-channel feature map to obtain a 1*512 sketch feature vector.

[0038] Step 3: Sketch point feature extraction and voxelization. Extract the geometric information of each point on the strokes of the sketch, and voxelize the sketch points based on the geometric information to obtain the voxel features of the points.

[0039] In this process, the geometric features of the sketch points are extracted manually, including coordinate information, gradient information, and Euclidean distance from the centroid. Each point is then converted into a voxel, and the geometric features of the points are converted into voxel features.

[0040] In this embodiment of the invention, sketch geometric information, etc., are as follows: Figure 2 As shown, threshold segmentation is used to extract all points on the sketch strokes, and the grayscale value g and (x, y) coordinate information of each point on the sketch image are recorded, while the z value is set to 0. Then, the centroid (x, y) is calculated using the recorded (x, y) position information of all points. c ,y c), center of mass (x) c ,y c The value is the average of all points (x, y). The distances of each point from the centroid along the x and y axes are used as two feature values ​​for that point. The distances of each point from the centroid in the x and y directions represent the overall compactness of the sketch. Based on the position of each point in the sketch image, the gradients (gradx, grady) of that point along the x and y axes are calculated. The gradient information (gradx, grady) in the x and y directions of each point indicates whether that point is on a stroke edge. The above information constitutes the geometric information (x, y, 0, g, xx) of each point in the sketch. c yy c (gradx, grady). Then, the geometric information of each point is manually extracted and voxelized, with each point converted into a voxel, such as... Figure 3 As shown, the point features are converted into voxel features, which helps Sketch-SparseVoxelNet to better extract the spatial features of the sketch and achieve more accurate sketch recognition.

[0041] Step 4: Voxel feature extraction from the sketch. For example... Figure 4 As shown, based on the Sparse 3D Convolutional Neural Network (S3DCB) module, a voxel feature extraction network Sketch-SparseVoxelNet is built. The transformed voxels are input into the voxel feature extraction network Sketch-SparseVoxelNet to extract sketch space features and obtain the feature vector of the sketch.

[0042] In this embodiment of the invention, nine sparse 3D convolutional neural network modules (S3DCB) are stacked to build a voxel feature extraction network Sketch-SparseVoxelNet, and voxel downsampling is performed step by step. The extracted features are averaged to obtain a 1*512 feature vector of the sparse sketch.

[0043] like Figure 5 As shown, by using three-dimensional sparse convolution, large blank areas of the sketch can be ignored, and the spatial relationship of the stroke points of the sketch can be extracted directly, as shown in formula (1).

[0044]

[0045] Among them, w ijz This represents the weights at position (x, y, z) of the convolution kernel, where k represents the kernel size, and x... ijz f represents the eigenvalues ​​of the voxel feature map. Sparse 3DConv Features are extracted only at the stroke points of the sketch, ignoring large blank areas of the sketch.

[0046] The 3D sparse convolution module (S3DCB) is composed of a basic 3D sparse convolution module (BS3DConv), 3D sparse convolution, and batch regularization stacked together, such as... Figure 6a , Figure 6b As shown, this effectively utilizes 3D sparse convolution to extract spatial relationships in sketches. Simultaneously, by incorporating the characteristics of 2D residual modules and introducing residual connections, it accelerates sketch model convergence and improves sketch recognition accuracy by continuously stacking modules to better extract sketch features. Sketch-SparseVoxelNet consists of nine stacked sparse 3D convolutional modules, and its detailed structure is shown in Table 2.

[0047] Step 5: Target Category Classification. The feature vectors generated by CNN and Sketch-SparseNet are fused and fed into the Sketch-SparseNet classifier for target category classification.

[0048] In this embodiment of the invention, the 1*512 feature vectors generated by CNN and Sketch-SparseVoxelNet are fused together to obtain a 1*512 feature vector, which is then fed into the Sketch-SparseNet classifier to output the scores for each category. The category with the highest score is the target category.

[0049] To verify the effectiveness of this method, the QuickDraw-414k dataset, a publicly available dataset, was used. Experimental results show that, for comparison with state-of-the-art networks, the method of this invention utilizes the QuickDraw-414k dataset. The QuickDraw-414k dataset contains data across 345 categories, including animals (cats, dogs), animation, and toys. Each category in the QuickDraw-414k dataset includes 1000 training sketches, 100 validation sketches, and 100 test sketches, totaling 414,000 sketches. The experimental results are shown in Table 1.

[0050] Table 1. Comparison of Sketch-SparseNet based on different advanced convolutional neural networks on QuickDraw-414k sketch recognition accuracy.

[0051]

[0052] Table 2 Sketch-SparseNet Architecture Table

[0053]

[0054]

[0055] Corresponding to the aforementioned embodiments of the sketch recognition method based on the fusion of three-dimensional sparse convolution and two-dimensional convolution, the present invention also provides embodiments of a sketch recognition device based on the fusion of three-dimensional sparse convolution and two-dimensional convolution.

[0056] See Figure 7 The sketch recognition device based on the fusion of three-dimensional sparse convolution and two-dimensional convolution provided in this embodiment of the invention includes a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the sketch recognition method based on the fusion of three-dimensional sparse convolution and two-dimensional convolution in the above embodiment.

[0057] The sketch recognition device based on the fusion of 3D sparse convolution and 2D convolution of this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 7 The diagram shown is a hardware structure diagram of any data processing-capable device, including the sketch recognition device based on the fusion of three-dimensional sparse convolution and two-dimensional convolution of this invention. (Except for...) Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0058] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0059] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0060] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the sketch recognition method based on the fusion of three-dimensional sparse convolution and two-dimensional convolution described in the above embodiments.

[0061] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0062] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sketch recognition method based on the fusion of three-dimensional sparse convolution and two-dimensional convolution, characterized in that... Includes the following steps: Step 1: Obtain the sketch image; Step 2: Extract features from the sketch image using 2D convolution to obtain the sketch image features; Step 3: Sketch point feature extraction and voxelization; Extract the geometric information of each point on the stroke of the sketch, and voxelize the sketch points based on the geometric information to obtain the voxels of the points; Step 4: Extract features from voxels at each point of the sketch to obtain sketch voxel features; Based on the voxels corresponding to the stroke points of the sketch segmentation, a first hash table is constructed. The three-dimensional sparse convolutional neural network extracts the voxel features of the current stroke point and the voxel features of the current stroke point and other stroke points through the first hash table. A second hash table is constructed based on the voxel features. The spatial relationship between the voxel features corresponding to each stroke point is constructed based on the second hash table, and finally the sketch voxel features are obtained. Step 5: Sketch classification; fuse sketch image features and sketch voxel features, and use a classifier to classify and recognize sketches.

2. The sketch recognition method based on the fusion of three-dimensional sparse convolution and two-dimensional convolution as described in claim 1, characterized in that: In step 3, threshold segmentation is used to extract points on the sketch strokes, and the gray values ​​and position information of each point on the sketch image are obtained. The centroid of the sketch is calculated through the position information. The gray values, position information, distances between each point and the centroid, and position gradients of each point are used as the geometric information of each point, and the geometric information is voxelized.

3. The sketch recognition method based on the fusion of three-dimensional sparse convolution and two-dimensional convolution according to claim 2, characterized in that: The position information refers to the coordinates along the x-axis and y-axis of the sketch plane, and the centroid is the average value of the coordinates of all points.

4. The sketch recognition method based on the fusion of three-dimensional sparse convolution and two-dimensional convolution according to claim 1, characterized in that: In step 4, a voxel feature extraction network is constructed based on a set of three-dimensional sparse convolutional neural networks to extract features from the transformed voxels and perform voxel downsampling step by step. The extracted features are averaged to obtain the voxel feature vector of the sparse sketch.

5. The sketch recognition method based on the fusion of three-dimensional sparse convolution and two-dimensional convolution according to claim 4, characterized in that: The three-dimensional sparse convolutional neural network extracts features only from the positions of sketch strokes, as shown in the following formula: in, This represents the weights of the convolution kernel at the corresponding positions on the three axes. Indicates the kernel size. The eigenvalues ​​represent the voxel feature maps.

6. The sketch recognition method based on the fusion of three-dimensional sparse convolution and two-dimensional convolution according to claim 4, characterized in that: The three-dimensional sparse convolutional neural network includes a basic three-dimensional sparse convolution module, three-dimensional sparse convolution, batch regularization, and also introduces residual connections.

7. The sketch recognition method based on the fusion of three-dimensional sparse convolution and two-dimensional convolution according to claim 1, characterized in that: In step 2, sketch image features are extracted using a basic convolutional neural network, and image downsampling is performed step by step to obtain a high-dimensional feature map of the sketch. Global average pooling is then performed on the high-dimensional feature map to obtain the sketch feature vector.

8. The sketch recognition method based on the fusion of three-dimensional sparse convolution and two-dimensional convolution according to claim 1, characterized in that: In step 5, a loss is constructed based on sketch image features, sketch voxel features and their fused features, a classification and recognition model is trained, and the trained classification and recognition model is used to classify and recognize the sketch image.

9. A sketch recognition device based on the fusion of three-dimensional sparse convolution and two-dimensional convolution, characterized in that, The method includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the sketch recognition method based on the fusion of three-dimensional sparse convolution and two-dimensional convolution as described in any one of claims 1-8.

Images