Classification Method, Device, Equipment and Storage Medium for Hand-Drawn Sketches

Abstract

The present invention relates to the technical field of image recognition, and discloses a classification method, device, equipment and storage medium for hand-drawn sketches. The method includes: extracting features of the hand-drawn sketch to be recognized through a target dilated convolution to obtain multi-scale stroke structures and blank image regions; densely extracting the hand-drawn sketch to be recognized through a target neural network to obtain the spatial density of the hand-drawn sketch; recognizing the multi-scale stroke structures and blank image regions in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the design type of the hand-drawn sketch to be recognized; classifying the hand-drawn sketch to be recognized according to the design type; through the above method, extracting the hand-drawn sketch to be recognized respectively by the target dilated convolution and the target neural network, then splicing in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the contour of the hand-drawn sketch to be recognized, and then classifying according to the design type corresponding to the contour, which can effectively improve the accuracy of classifying hand-drawn sketches.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and particularly to a classification method, device, equipment and storage medium for hand-drawn sketches. Background Art

[0002] Hand-drawn sketches have been an important means of communication between people since ancient times. With the development of electronic technology and the improvement of the intelligent level, people's demand for interacting with electronic devices through sketches is also increasing, making the recognition of hand-drawn sketches gradually become a research hotspot in the field of computer applications. Different from general color images, hand-drawn sketches have high semantic abstraction, structural diversity, and unique stroke sparsity, and lack texture and brightness information. Currently, the deep learning models used for recognizing hand-drawn sketches are mainly based on convolutional neural networks. However, most convolutional neural network models are designed for the characteristics of general natural images. If a convolutional neural network model is forcibly used to recognize and classify hand-drawn sketches, the types of recognized hand-drawn sketches will be inaccurate, resulting in low accuracy of classified hand-drawn sketches.

[0003] The above content is only used to assist in understanding the technical solution of the present invention, and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of the present invention is to provide a classification method, device, equipment and storage medium for hand-drawn sketches, aiming to solve the technical problem of low accuracy in classifying hand-drawn sketches in the prior art.

[0005] To achieve the above purpose, the present invention provides a classification method for hand-drawn sketches, and the classification method for hand-drawn sketches includes the following steps:

[0006] Performing feature extraction on the hand-drawn sketch to be recognized through a target dilated convolution to obtain a multi-scale stroke structure and a blank image area;

[0007] Performing dense extraction on the hand-drawn sketch to be recognized through a target neural network to obtain the spatial density of the hand-drawn sketch;

[0008] Identifying the multi-scale stroke structure and the blank image area in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the design type of the hand-drawn sketch to be recognized;

[0009] Classifying the hand-drawn sketch to be recognized according to the design type.

[0010] Optionally, the performing feature extraction on the hand-drawn sketch to be recognized through a target dilated convolution to obtain a multi-scale stroke structure and a blank image area includes:

[0011] Detect the hand-drawn sketch to be recognized to obtain the overlapping area of the hand-drawn sketch;

[0012] Extract features from the overlapping area of the hand-drawn sketch through the target dilated convolution to obtain the stroke structure of the overlapping area and the stroke structure of the non-overlapping area;

[0013] Fuse the stroke structure of the overlapping area and the stroke structure of the non-overlapping area at different scales to obtain the multi-scale stroke structure;

[0014] Perform edge search on the multi-scale stroke structure to obtain the blank image area.

[0015] Optionally, the extracting features from the overlapping area of the hand-drawn sketch through the target dilated convolution to obtain the stroke structure of the overlapping area and the stroke structure of the non-overlapping area includes:

[0016] Extract features from the overlapping area of the hand-drawn sketch through the target dilated convolution to obtain the direction feature, the corner feature, and the line segment endpoints;

[0017] Linearly fit the line segment endpoints according to the direction feature and the corner feature to obtain a target number of segmented curves;

[0018] Extract the stroke image depth of the target number of segmented curves;

[0019] Screen the segmented curves according to the stroke image depth to obtain the target depth segmented curves;

[0020] Generate the stroke structure of the overlapping area according to the target depth segmented curves;

[0021] Perform depth matching on the target depth segmented curves and the hand-drawn sketch to be recognized;

[0022] Generate the stroke structure of the non-overlapping area according to the matching result.

[0023] Optionally, the performing edge search on the multi-scale stroke structure to obtain the blank image area includes:

[0024] Select a target starting point on the multi-scale stroke structure;

[0025] Determine multiple search directions according to the preset search angle;

[0026] Start from the target starting point and perform edge search on the multi-scale stroke structure in each search direction simultaneously;

[0027] If a stroke edge is encountered during the edge search, mark the position of the stroke edge;

[0028] Determine the region of the multi-scale stroke structure according to the marked position;

[0029] Obtain the blank image region according to the region of the multi-scale stroke structure and the region where the hand-drawn sketch to be recognized is located.

[0030] Optionally, the intensive extraction of the hand-drawn sketch to be recognized by the target neural network to obtain the spatial density of the hand-drawn sketch includes:

[0031] Cut the hand-drawn sketch to be recognized to obtain non-overlapping hand-drawn sketch blocks;

[0032] Count the number of the non-overlapping hand-drawn sketch blocks;

[0033] When the number is greater than the preset number threshold, perform intensive extraction on the non-overlapping hand-drawn sketch blocks in sequence by the target neural network according to the preset order relationship to obtain the density of each sketch block;

[0034] Connect the hand-drawn sketch blocks within the target range according to the density of each sketch block to obtain the target hand-drawn sketch block;

[0035] Determine the spatial density of the hand-drawn sketch according to the position coordinates of the target hand-drawn sketch block.

[0036] Optionally, the recognition of the multi-scale stroke structure and the blank image region in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the design type of the hand-drawn sketch to be recognized includes:

[0037] Stitch the multi-scale stroke structure and the blank image region in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the current hand-drawn sketch contour;

[0038] Perform oblique floating and deepening on the current hand-drawn sketch contour;

[0039] Recognize the current hand-drawn sketch contour after oblique floating and deepening through the sketch design strategy to obtain the design type of the hand-drawn sketch to be recognized.

[0040] Optionally, after classifying the hand-drawn sketch to be recognized according to the design type, it further includes:

[0041] Obtain the name and size of the hand-drawn sketch to be recognized;

[0042] Upload the hand-drawn sketch to be recognized to the target hand-drawn sketch database according to the name, size, and the current network running state;

[0043] After the upload is successful, update the sketch design strategy according to the target hand-drawn sketch database.

[0044] In addition, to achieve the above object, the present invention further provides a classification device for hand-drawn sketches, and the classification device for hand-drawn sketches includes:

[0045] An acquisition module, configured to acquire the current network information of a target terminal device;

[0046] A connection module, configured to establish a connection with the target terminal device through a peer-to-peer interconnection network policy when the current network information and the target network information are not in the same local area network;

[0047] A receiving module, configured to receive a control instruction sent by the target terminal device when the connection with the target terminal device is successful;

[0048] A control module, configured to perform screen mirroring on the content to be screen-mirrored through the control instruction, so as to implement control of screen mirroring based on the terminal device.

[0049] In addition, to achieve the above object, the present invention further provides a classification device for hand-drawn sketches, and the classification device for hand-drawn sketches includes: a memory, a processor, and a classification program for hand-drawn sketches stored on the memory and executable on the processor, and the classification program for hand-drawn sketches is configured to implement the classification method for hand-drawn sketches as described above.

[0050] In addition, to achieve the above object, the present invention further provides a storage medium, on which a classification program for hand-drawn sketches is stored, and when the classification program for hand-drawn sketches is executed by a processor, the classification method for hand-drawn sketches as described above is implemented.

[0051] The classification method for hand-drawn sketches provided by the present invention extracts features of the hand-drawn sketch to be recognized through a target dilated convolution to obtain a multi-scale stroke structure and a blank image area; densely extracts the hand-drawn sketch to be recognized through a target neural network to obtain the spatial density of the hand-drawn sketch; recognizes the multi-scale stroke structure and the blank image area in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the design type of the hand-drawn sketch to be recognized; classifies the hand-drawn sketch to be recognized according to the design type; through the above method, extracts the hand-drawn sketch to be recognized through a target dilated convolution and a target neural network respectively, then splices them in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the contour of the hand-drawn sketch to be recognized, and then classifies according to the design type corresponding to the contour, which can effectively improve the accuracy of classifying hand-drawn sketches. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a schematic structural diagram of a classification device for hand-drawn sketches in a hardware operating environment related to the solution of an embodiment of the present invention;

[0053] Figure 2Schematic flowchart of the first embodiment of the classification method for hand-drawn sketches of the present invention;

[0054] Figure 3 Schematic diagram of a hand-drawn sketch to be recognized in an embodiment of the classification method for hand-drawn sketches of the present invention;

[0055] Figure 4 Schematic flowchart of the second embodiment of the classification method for hand-drawn sketches of the present invention;

[0056] Figure 5 Schematic flowchart of the third embodiment of the classification method for hand-drawn sketches of the present invention;

[0057] Figure 6 Schematic diagram of functional modules of the first embodiment of the classification device for hand-drawn sketches of the present invention.

[0058] The realization, functional features and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0059] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0060] Refer to Figure 1 , Figure 1 Schematic diagram of the structure of the classification device for hand-drawn sketches in the hardware operating environment related to the embodiment solution of the present invention.

[0061] As Figure 1 shown, the classification device for hand-drawn sketches may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and optionally the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless-fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0062] Those skilled in the art can understand, Figure 1The structure shown does not constitute a limitation on the classification device for hand-drawn sketches, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0063] As Figure 1 shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and a classification program for hand-drawn sketches.

[0064] In Figure 1 the classification device for hand-drawn sketches shown, the network interface 1004 is mainly used for data communication with the network integrated platform workstation; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the classification device for hand-drawn sketches of the present invention may be arranged in the classification device for hand-drawn sketches. The classification device for hand-drawn sketches calls the classification program for hand-drawn sketches stored in the memory 1005 through the processor 1001 and executes the classification method for hand-drawn sketches provided by the embodiments of the present invention.

[0065] Based on the above hardware structure, an embodiment of the classification method for hand-drawn sketches of the present invention is proposed.

[0066] Referring to Figure 2 , Figure 2 is a schematic flowchart of the first embodiment of the classification method for hand-drawn sketches of the present invention.

[0067] In the first embodiment, the classification method for hand-drawn sketches includes the following steps:

[0068] Step S10, extracting features from the hand-drawn sketch to be recognized through a target dilated convolution to obtain a multi-scale stroke structure and a blank image area.

[0069] It should be noted that the execution subject of this embodiment is the classification device for hand-drawn sketches, and may also be other devices that can achieve the same or similar functions, such as a sketch classifier, etc. This embodiment does not limit this. In this embodiment, a sketch classifier is used as an example for illustration.

[0070] It should be understood that the hand-drawn sketch to be recognized refers to the hand-drawn sketch that needs to be recognized and classified. Refer to Figure 3 , Figure 3It is a schematic diagram of a hand-drawn sketch to be recognized. The overall hand-drawn sketch to be recognized in this embodiment is relatively complex, including the thickness of lines, the outline of shapes, and the overlap of content, etc. Dilated convolution refers to a convolution that can keep the size of the output feature map unchanged while increasing the receptive field. The difference between dilated convolution and ordinary convolution lies in the introduction of the "dilation rate". This parameter defines the spacing of each value when the convolution kernel processes data. Specifically, it means that convolution units with a weight of 0 are filled in the convolution kernel. Compared with the original ordinary convolution, dilated convolution has an additional hyperparameter, that is, the number of intervals of effective convolution units, while the dilation rate of an ordinary convolution kernel is 1 and there is no 0 padding. When multiple convolution kernels with different dilation rates are combined, it is equivalent to combining convolution kernels with different receptive fields to obtain the target dilated convolution, and then the target dilated convolution is used to extract features from the hand-drawn sketch to be recognized.

[0071] It can be understood that the multi-scale stroke structure refers to the structure composed of each stroke in the hand-drawn sketch to be recognized. Since there are differences in the thickness and shade of each stroke, therefore, in the process of using dilated convolution to extract features from the hand-drawn sketch to be recognized, it is necessary to divide the hand-drawn sketch to be recognized into hand-drawn sketches of different scales for feature extraction to obtain the multi-scale stroke structure. The blank image area refers to the blank area in the hand-drawn sketch to be recognized, and this blank area includes the blank area between the stroke structures and the blank area on the outermost side of the hand-drawn sketch to be recognized.

[0072] Step S20: Densely extract the hand-drawn sketch to be recognized through the target neural network to obtain the spatial density of the hand-drawn sketch.

[0073] It can be understood that the spatial density of the hand-drawn sketch refers to the density of sketch blocks in the hand-drawn sketch to be recognized. Since dilated convolution has no correlation between the information obtained by distant convolutions, which affects the classification of the final hand-drawn sketch to be recognized, therefore, it is necessary to use a target neural network with the same size receptive field to densely extract the hand-drawn sketch to be recognized.

[0074] Furthermore, step S20 includes: cutting the hand-drawn sketch to be recognized to obtain non-overlapping hand-drawn sketch blocks; counting the number of the non-overlapping hand-drawn sketch blocks; when the number is greater than a preset number threshold, densely extract the non-overlapping hand-drawn sketch blocks in sequence through the target neural network according to a preset order relationship to obtain the density of each sketch block; connect the hand-drawn sketch blocks within the target range according to the density of each sketch block to obtain the target hand-drawn sketch block; determine the spatial density of the hand-drawn sketch according to the position coordinates of the target hand-drawn sketch block.

[0075] It should be understood that after obtaining the hand-drawn sketch to be recognized, the hand-drawn sketch to be recognized is cut into hand-drawn sketch blocks with the target number that do not overlap each other in a slicing manner. The preset quantity threshold refers to the minimum quantity for judging whether an image block is dense, and the preset sequence relationship refers to the relationship of densely extracting hand-drawn sketch blocks. This preset sequence relationship can be clockwise or counterclockwise. Through the preset sequence relationship, it can be ensured that non-overlapping hand-drawn sketch blocks will not be repeatedly and densely extracted.

[0076] It can be understood that the target hand-drawn sketch block refers to the operation block obtained by connecting the hand-drawn sketch blocks within the target range. After obtaining the density of each sketch block, the hand-drawn sketch blocks within the target range are determined according to the density of each sketch block. After the connection is completed, the spatial density of the hand-drawn sketch is determined according to the position coordinates where the target hand-drawn sketch block is currently located.

[0077] Step S30: Identify the multi-scale stroke structure and the blank image area in the channel dimension according to the spatial density of the hand-drawn sketch, and obtain the design type of the hand-drawn sketch to be recognized.

[0078] It should be understood that the design type refers to the type of hand-drawn sketch by the painter or designer. This design type includes conceptual hand-drawn sketch type, thinking hand-drawn sketch type, technical hand-drawn sketch type, report hand-drawn sketch type, and emotional hand-drawn sketch type. After obtaining the spatial density of the hand-drawn sketch, the multi-scale stroke structure and the blank image area are spliced in the channel dimension according to the spatial density of the hand-drawn sketch. After the splicing is completed, the current contour of the hand-drawn sketch obtained by splicing is then recognized to obtain the design type of the hand-drawn sketch to be recognized.

[0079] Step S40: Classify the hand-drawn sketch to be recognized according to the design type.

[0080] It can be understood that after obtaining the design type, the hand-drawn sketch to be recognized is classified into the corresponding hand-drawn sketch library according to the design type. For example, when the design type is the conceptual hand-drawn sketch type, the hand-drawn sketch to be recognized is classified into the conceptual hand-drawn sketch database.

[0081] Further, after step S40, it further includes: obtaining the name and size of the hand-drawn sketch to be recognized; uploading the hand-drawn sketch to be recognized to the target hand-drawn sketch database according to the name, size, and the current network operation status; after the upload is successful, updating the sketch design strategy according to the target hand-drawn sketch database.

[0082] It should be understood that the name refers to the name given by the painter or designer when hand-drawing the hand-drawn sketch to be recognized. For example, the hand-drawn sketch of an office building, version V1. The size refers to the size of the hand-drawn sketch to be recognized. For example, in reality, the size of the hand-drawn sketch to be recognized is: 80 cm in length and 60 cm in width. In an electronic device, the size of the hand-drawn sketch to be recognized is: 2240 px in length and 1680 px in width.

[0083] It can be understood that before uploading the hand-drawn sketch to be recognized to the target hand-drawn sketch database, it is necessary to obtain the current network running state of the sketch classifier. When the current network running state is relatively good, the hand-drawn sketch to be recognized is directly uploaded to the target hand-drawn sketch database in the form of the original image. When the current network running state is relatively poor, the hand-drawn sketch to be recognized is compressed. After the compression is completed, the compressed hand-drawn sketch to be recognized is uploaded to the target hand-drawn sketch database, and then the sketch design strategy is updated in real time to facilitate the recognition of other similar hand-drawn sketches.

[0084] In this embodiment, the target atrous convolution is used to extract features from the hand-drawn sketch to be recognized, obtaining a multi-scale stroke structure and a blank image area; the target neural network is used to densely extract the hand-drawn sketch to be recognized, obtaining the spatial density of the hand-drawn sketch; the multi-scale stroke structure and the blank image area are recognized in the channel dimension according to the spatial density of the hand-drawn sketch, obtaining the design type of the hand-drawn sketch to be recognized; the hand-drawn sketch to be recognized is classified according to the design type; in the above manner, the target atrous convolution and the target neural network are respectively used to extract the hand-drawn sketch to be recognized, and then they are spliced in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the contour of the hand-drawn sketch to be recognized, and then classified according to the design type corresponding to the contour, which can effectively improve the accuracy of classifying hand-drawn sketches.

[0085] In one embodiment, as Figure 4 described, based on the first embodiment, the second embodiment of the classification method of the hand-drawn sketch of the present invention is proposed. The step S10 includes:

[0086] Step S101, detecting the hand-drawn sketch to be recognized to obtain the overlapping area of the hand-drawn sketch.

[0087] It should be understood that the overlapping area of the hand-drawn sketch refers to the overlapping area in the hand-drawn sketch to be recognized. For example, the hand-drawn sketch to be recognized is the hand-drawn sketch of an office building, and the hand-drawn sketch of the office building is divided into the hand-drawn sketch of the first unit and the hand-drawn sketch of the second unit. There is an overlapping part between the hand-drawn sketch of the first unit and the hand-drawn sketch of the second unit, and this overlapping part is the overlapping area of the hand-drawn sketch.

[0088] Step S102, perform feature extraction on the overlapping area of the hand-drawn sketch through a target dilated convolution to obtain the stroke structure of the overlapping area and the stroke structure of the non-overlapping area.

[0089] It can be understood that the stroke structure of the overlapping area refers to the stroke structure of the overlapping area in the hand-drawn sketch to be recognized. Similarly, the stroke structure of the non-overlapping area refers to the stroke structure of the non-overlapping area in the hand-drawn sketch to be recognized. After obtaining the overlapping area of the hand-drawn sketch, the features of the overlapping area of the hand-drawn sketch are extracted through a target dilated convolution respectively, and then the stroke structure of the overlapping area and the stroke structure of the non-overlapping area are obtained according to the target depth segmented curve.

[0090] Further, step S102 includes: performing feature extraction on the overlapping area of the hand-drawn sketch through a target dilated convolution to obtain direction features, corner features, and line segment endpoints; performing linear fitting on the line segment endpoints according to the direction features and the corner features to obtain a target number of segmented curves; extracting the stroke image depth of the target number of segmented curves; screening the segmented curves according to the stroke image depth to obtain a target depth segmented curve; generating a stroke structure of the overlapping area according to the target depth segmented curve; performing depth matching on the target depth segmented curve and the hand-drawn sketch to be recognized; and generating a stroke structure of the non-overlapping area according to the matching result.

[0091] It should be understood that the direction feature refers to the direction feature point of the direction mutation of the stroke structure in the overlapping area of the hand-drawn sketch, the corner feature refers to the feature point with the largest angle in the stroke structure in the overlapping area of the hand-drawn sketch, and the line segment endpoints refer to the feature points at both ends of the line segment in the stroke structure in the overlapping area of the hand-drawn sketch. Then, linear fitting is performed on the line segment endpoints according to the direction features and the corner features. After the fitting is completed, a target number of segmented curves are obtained.

[0092] It can be understood that the target depth segmented curve refers to the curve with the highest stroke image depth among the target number of segmented curves. In a hand-drawn sketch, the higher the stroke image depth, the more ink is drawn at the same position. The reason for the more ink can be that the drawing is performed at this position multiple times or the drawing force is large. After obtaining the target depth segmented curve, a stroke structure of the overlapping area is generated according to the target depth segmented curve, and then the segmented curves with the same depth in the hand-drawn sketch to be recognized are matched. At this time, a stroke structure of the non-overlapping area is generated according to the remaining other segmented curves.

[0093] Step S103, fuse the stroke structure of the overlapping area and the stroke structure of the non-overlapping area at different scales to obtain a multi-scale stroke structure.

[0094] It should be understood that after obtaining the stroke structures in the overlapping regions and the stroke structures in the non-overlapping regions, the stroke structures in the overlapping regions and the stroke structures in the non-overlapping regions are divided according to the scale size, and then the stroke structures in the overlapping regions and the stroke structures in the non-overlapping regions with different scales are fused to obtain multi-scale stroke structures.

[0095] Step S104: Perform edge search on the multi-scale stroke structure to obtain a blank image area.

[0096] It should be understood that after obtaining the multi-scale stroke structure, edge search is performed on the multi-scale stroke structure. After the search is completed, the area formed by the search is the area of the multi-scale stroke structure, and the remaining area is the blank image area.

[0097] Furthermore, performing edge search on the multi-scale stroke structure to obtain a blank image area includes: selecting a target starting point on the multi-scale stroke structure; determining a plurality of search directions according to a preset search angle; starting from the target starting point, performing edge search on the multi-scale stroke structure simultaneously in each search direction; if a stroke edge is encountered during the edge search, marking the position of the stroke edge; determining the area of the multi-scale stroke structure according to the marked position; and obtaining the blank image area according to the area of the multi-scale stroke structure and the area where the hand-drawn sketch to be recognized is located.

[0098] It can be understood that the target starting point refers to the starting point for performing edge search on the multi-scale stroke structure. The target starting point can be located at the center position of the multi-scale stroke structure. The preset search angle refers to the angle for searching the multi-scale stroke structure. Through the preset search angle, the number of search directions can be determined. For example, if the search angle is 60°, the number of search directions is 6. After determining the search angle and search directions, edge search is performed on the multi-scale stroke structure simultaneously. If the stroke edge in a certain direction has been touched during the search, the position of the stroke edge in that direction is marked, and the search continues in other directions until the stroke edges in all search directions have been touched. After the search is completed, the area of the multi-scale stroke structure is obtained, and the area other than the area of the multi-scale stroke structure is used as the blank image area.

[0099] In this embodiment, the overlapping area of the hand-drawn sketch to be recognized is detected, and the overlapping area of the hand-drawn sketch is obtained; the overlapping area of the hand-drawn sketch is feature-extracted by the target dilated convolution to obtain the stroke structure of the overlapping area and the stroke structure of the non-overlapping area; the stroke structure of the overlapping area and the stroke structure of the non-overlapping area are fused according to different scales to obtain a multi-scale stroke structure; the multi-scale stroke structure is edge-searched to obtain a blank image area; in the above manner, the overlapping area of the hand-drawn sketch in the hand-drawn sketch to be recognized is detected, then the stroke structure of the overlapping area and the stroke structure of the non-overlapping area are extracted according to the target dilated convolution feature, then a multi-scale stroke structure is fused according to different scales, and finally the multi-scale stroke structure is edge-searched, so that the accuracy of obtaining the blank image area can be effectively improved.

[0100] In one embodiment, as Figure 5 described, based on the first embodiment, the third embodiment of the classification method of the hand-drawn sketch of the present invention is proposed, and the step S30 includes:

[0101] Step S301, the multi-scale stroke structure and the blank image area are spliced in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the current hand-drawn sketch contour.

[0102] It can be understood that after obtaining the multi-scale stroke structure and the blank image area, the multi-scale stroke structure and the blank image area are spliced in the channel dimension. Before splicing, it is necessary to judge whether the dimensions of the multi-scale stroke structure and the blank image area are consistent. If so, the multi-scale stroke structure and the blank image area are directly spliced in the channel dimension. If not, the multi-scale stroke structure and the blank image area are converted to the same dimension in the channel dimension. After the conversion is completed, the converted multi-scale stroke structure and the blank image area are spliced to obtain the current hand-drawn sketch contour.

[0103] Step S302, perform bevel embossing on the current hand-drawn sketch contour.

[0104] It should be understood that after obtaining the current hand-drawn sketch contour, in order to improve the accuracy of recognizing the current hand-drawn sketch contour, the contour of the current hand-drawn sketch contour needs to be enlarged. Specifically, bevel embossing is performed on the current hand-drawn sketch contour, which includes two processes. First, the embossing attribute is set on the current hand-drawn sketch contour. Second, the depth is increased outside the embossing and the current hand-drawn sketch contour, so that the features of the current hand-drawn sketch contour are more obvious.

[0105] Step S303, recognize the bevel-embossed current hand-drawn sketch contour through the sketch design strategy to obtain the design type of the hand-drawn sketch to be recognized.

[0106] It can be understood that the sketch design strategy refers to the design strategy of the painter or designer when drawing the hand-drawn sketch to be recognized. By using the sketch design strategy to recognize the current hand-drawn sketch contour after oblique embossing and deepening, the design type of the hand-drawn sketch to be recognized is determined according to the recognition result.

[0107] In this embodiment, the multi-scale stroke structure and the blank image area are spliced in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the current hand-drawn sketch contour; the current hand-drawn sketch contour is obliquely embossed and deepened; the sketch design strategy is used to recognize the current hand-drawn sketch contour after oblique embossing and deepening to obtain the design type of the hand-drawn sketch to be recognized; in the above manner, the multi-scale stroke structure and the blank image area are spliced in the channel dimension according to the spatial density of the hand-drawn sketch, then the relief attribute is set on the current hand-drawn sketch contour, and the depth is increased for the set current hand-drawn sketch contour, and finally the sketch design strategy is used for recognition, so as to effectively improve the accuracy of recognizing the design type of the hand-drawn sketch to be recognized.

[0108] In addition, an embodiment of the present invention also proposes a storage medium, on which a classification program for hand-drawn sketches is stored. When the classification program for hand-drawn sketches is executed by a processor, the steps of the classification method for hand-drawn sketches as described above are implemented.

[0109] Since this storage medium adopts all the technical solutions of the above all embodiments, it at least has all the beneficial effects brought by the technical solutions of the above embodiments, which will not be elaborated one by one here.

[0110] In addition, with reference to Figure 6 , an embodiment of the present invention also proposes a classification device for hand-drawn sketches, and the classification device for hand-drawn sketches includes:

[0111] A feature extraction module 10, configured to extract features from the hand-drawn sketch to be recognized through a target dilated convolution to obtain a multi-scale stroke structure and a blank image area.

[0112] A dense extraction module 20, configured to perform dense extraction on the hand-drawn sketch to be recognized through a target neural network to obtain the spatial density of the hand-drawn sketch.

[0113] An identification module 30, configured to identify the multi-scale stroke structure and the blank image area in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the design type of the hand-drawn sketch to be recognized.

[0114] A classification module 40, configured to classify the hand-drawn sketch to be recognized according to the design type.

[0115] In this embodiment, target dilated convolution is used to extract features from the hand-drawn sketch to be recognized, obtaining multi-scale stroke structures and blank image regions; target neural network is used to densely extract the hand-drawn sketch to be recognized, obtaining the spatial density of the hand-drawn sketch; the multi-scale stroke structures and the blank image regions are recognized in the channel dimension according to the spatial density of the hand-drawn sketch, obtaining the design type of the hand-drawn sketch to be recognized; the hand-drawn sketch to be recognized is classified according to the design type; in the above manner, by separately extracting the hand-drawn sketch to be recognized using target dilated convolution and target neural network, then splicing in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the contour of the hand-drawn sketch to be recognized, and then classifying according to the design type corresponding to the contour, the accuracy of classifying hand-drawn sketches can be effectively improved.

[0116] It should be noted that the above-described work process is only illustrative and does not limit the protection scope of the present invention. In actual applications, those skilled in the art can select some or all of them according to actual needs to achieve the purpose of the solution of this embodiment, and no limitation is made here.

[0117] In addition, for the technical details not described in detail in this embodiment, reference can be made to the hand-drawn sketch classification method provided in any embodiment of the present invention, and details are not described herein again.

[0118] In one embodiment, the feature extraction module 10 is further configured to detect the hand-drawn sketch to be recognized, obtaining the overlapping region of the hand-drawn sketch; using target dilated convolution to extract features from the overlapping region of the hand-drawn sketch, obtaining the stroke structure of the overlapping region and the stroke structure of the non-overlapping region; fusing the stroke structure of the overlapping region and the stroke structure of the non-overlapping region according to different scales, obtaining multi-scale stroke structures; performing edge search on the multi-scale stroke structures to obtain blank image regions.

[0119] In one embodiment, the feature extraction module 10 is further configured to use target dilated convolution to extract features from the overlapping region of the hand-drawn sketch, obtaining direction features, corner features, and line segment endpoints; linearly fitting the line segment endpoints according to the direction features and the corner features, obtaining a target number of segmented curves; extracting the stroke image depth of the target number of segmented curves; screening the segmented curves according to the stroke image depth, obtaining target depth segmented curves; generating the stroke structure of the overlapping region according to the target depth segmented curves; performing depth matching on the target depth segmented curves and the hand-drawn sketch to be recognized; generating the stroke structure of the non-overlapping region according to the matching result.

[0120] In one embodiment, the feature extraction module 10 is further configured to select a target starting point on the multi-scale stroke structure; determine a plurality of search directions according to a preset search angle; perform edge search on the multi-scale stroke structure simultaneously in each search direction starting from the target starting point; if a stroke edge is encountered during the edge search, mark the position of the stroke edge; determine the area of the multi-scale stroke structure according to the marked positions; and obtain a blank image area based on the area of the multi-scale stroke structure and the area where the hand-drawn sketch to be recognized is located.

[0121] In one embodiment, the dense extraction module 20 is further configured to cut the hand-drawn sketch to be recognized to obtain non-overlapping hand-drawn sketch blocks; count the number of the non-overlapping hand-drawn sketch blocks; when the number is greater than a preset number threshold, perform dense extraction on the non-overlapping hand-drawn sketch blocks in sequence according to a preset order relationship through a target neural network to obtain the density of each sketch block; connect the hand-drawn sketch blocks within a target range according to the density of each sketch block to obtain a target hand-drawn sketch block; and determine the spatial density of the hand-drawn sketch according to the position coordinates of the target hand-drawn sketch block.

[0122] In one embodiment, the recognition module 30 is further configured to splice the multi-scale stroke structure and the blank image area in the channel dimension according to the spatial density of the hand-drawn sketch to obtain a current hand-drawn sketch contour; perform oblique floating and deepening on the current hand-drawn sketch contour; and recognize the current hand-drawn sketch contour after oblique floating and deepening through a sketch design strategy to obtain the design type of the hand-drawn sketch to be recognized.

[0123] In one embodiment, the classification module 40 is further configured to obtain the name and size of the hand-drawn sketch to be recognized; upload the hand-drawn sketch to be recognized to a target hand-drawn sketch database according to the name, size, and the current network running state; and update the sketch design strategy according to the target hand-drawn sketch database after the upload is successful.

[0124] For other embodiments or implementation methods of the hand-drawn sketch classification device of the present invention, reference may be made to the above method embodiments, and details are not repeated here.

[0125] In addition, it should be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or system including the element.

[0126] The serial numbers of the embodiments of the present invention above are only for description and do not represent the superiority or inferiority of the embodiments.

[0127] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as a Read Only Memory (ROM) / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, an integrated platform workstation, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0128] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of the present invention.

Claims

1. A classification method for hand-drawn sketches, characterized in that, The classification method of the hand-drawn sketch includes the following steps: Extract features from the hand-drawn sketch to be recognized through the target dilated convolution, and obtain multi-scale stroke structures and blank image regions; Densely extract the hand-drawn sketch to be recognized through the target neural network to obtain the spatial density of the hand-drawn sketch; Identify the multi-scale stroke structures and the blank image regions in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the design type of the hand-drawn sketch to be recognized; Classify the hand-drawn sketch to be recognized according to the design type; The step of extracting features from the hand-drawn sketch to be recognized through the target dilated convolution to obtain multi-scale stroke structures and blank image regions includes: Detect the hand-drawn sketch to be recognized to obtain the overlapping region of the hand-drawn sketch; where the overlapping region of the hand-drawn sketch refers to the overlapping region in the hand-drawn sketch to be recognized; Extract features from the overlapping region of the hand-drawn sketch through the target dilated convolution to obtain the stroke structure of the overlapping region and the stroke structure of the non-overlapping region; Fuse the stroke structure of the overlapping region and the stroke structure of the non-overlapping region according to different scales to obtain multi-scale stroke structures; Perform edge search on the multi-scale stroke structures to obtain blank image regions; The step of densely extracting the hand-drawn sketch to be recognized through the target neural network to obtain the spatial density of the hand-drawn sketch includes: Cut the hand-drawn sketch to be recognized to obtain non-overlapping hand-drawn sketch blocks; Count the number of the non-overlapping hand-drawn sketch blocks; When the number is greater than the preset number threshold, densely extract the non-overlapping hand-drawn sketch blocks in sequence through the target neural network according to the preset order relationship to obtain the density of each sketch block; Connect the hand-drawn sketch blocks within the target range according to the density of each sketch block to obtain the target hand-drawn sketch block; Determine the spatial density of the hand-drawn sketch according to the position coordinates of the target hand-drawn sketch block.

2. The classification method of the hand-drawn sketch according to claim 1, characterized in that, The step of extracting features from the overlapping region of the hand-drawn sketch through the target dilated convolution to obtain the stroke structure of the overlapping region and the stroke structure of the non-overlapping region includes: Extract features from the overlapping region of the hand-drawn sketch through the target dilated convolution to obtain direction features, corner features, and line segment endpoints; Perform linear fitting on the line segment endpoints according to the direction features and the corner features to obtain a target number of segmented curves; Extract the stroke image depth of the target number of segmented curves; Screen the segmented curves according to the stroke image depth to obtain target depth segmented curves; Generate the stroke structure of the overlapping region according to the target depth segmented curves; Perform depth matching on the target depth segmented curves and the hand-drawn sketch to be recognized; Generate the stroke structure of the non-overlapping region according to the matching result.

3. The classification method of the hand-drawn sketch according to claim 1, characterized in that, The step of performing edge search on the multi-scale stroke structures to obtain blank image regions includes: Select a target starting point on the multi-scale stroke structures; Determine multiple search directions according to the preset search angle; Start from the target starting point and perform edge search on the multi-scale stroke structures simultaneously in each search direction; If a stroke edge is encountered during the edge search process, mark the position of the stroke edge; Determine the region of the multi-scale stroke structure based on the marked positions; Obtain the blank image region based on the region of the multi-scale stroke structure and the region where the hand-drawn sketch to be recognized is located.

4. The classification method of hand-drawn sketches according to claim 1, characterized in that, Performing recognition on the multi-scale stroke structure and the blank image region in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the design type of the hand-drawn sketch to be recognized, including: Stitch the multi-scale stroke structure and the blank image region in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the contour of the current hand-drawn sketch; Perform oblique floating and darkening on the contour of the current hand-drawn sketch; Identify the design type of the hand-drawn sketch to be recognized by recognizing the contour of the current hand-drawn sketch after oblique floating and darkening through the sketch design strategy.

5. The classification method for hand-drawn sketches according to any one of claims 1 to 4, characterized in that, After classifying the hand-drawn sketch to be recognized according to the design type, it further includes: Obtain the name and size of the hand-drawn sketch to be recognized; Upload the hand-drawn sketch to be recognized to the target hand-drawn sketch database according to the name, size, and the current network running state; After successful upload, update the sketch design strategy according to the target hand-drawn sketch database.

6. A classification device for hand-drawn sketches, characterized in that, The classification device for the hand-drawn sketch includes: A feature extraction module for extracting features from the hand-drawn sketch to be recognized through a target dilated convolution to obtain a multi-scale stroke structure and a blank image region; A dense extraction module for densely extracting the hand-drawn sketch to be recognized through a target neural network to obtain the spatial density of the hand-drawn sketch; An identification module for performing identification on the multi-scale stroke structure and the blank image region in the channel dimension according to the spatial density of the hand-drawn sketch to obtain the design type of the hand-drawn sketch to be recognized; A classification module for classifying the hand-drawn sketch to be recognized according to the design type; The feature extraction module is further configured to detect the hand-drawn sketch to be recognized to obtain the overlapping region of the hand-drawn sketch; where the overlapping region of the hand-drawn sketch refers to the overlapping region in the hand-drawn sketch to be recognized; extract features from the overlapping region of the hand-drawn sketch through a target dilated convolution to obtain the stroke structure of the overlapping region and the stroke structure of the non-overlapping region; fuse the stroke structure of the overlapping region and the stroke structure of the non-overlapping region according to different scales to obtain a multi-scale stroke structure; perform edge search on the multi-scale stroke structure to obtain a blank image region; The dense extraction module is further configured to cut the hand-drawn sketch to be recognized to obtain non-overlapping hand-drawn sketch blocks; count the number of the non-overlapping hand-drawn sketch blocks; when the number is greater than a preset number threshold, densely extract the non-overlapping hand-drawn sketch blocks in sequence through a target neural network according to a preset order relationship to obtain the density of each sketch block; connect the hand-drawn sketch blocks within the target range according to the density of each sketch block to obtain a target hand-drawn sketch block; determine the spatial density of the hand-drawn sketch according to the position coordinates of the target hand-drawn sketch block.

7. A classification device for hand-drawn sketches, characterized in that, The classification device for the hand-drawn sketches includes: a memory, a processor, and a classification program for hand-drawn sketches stored on the memory and executable on the processor, where the classification program for hand-drawn sketches is configured to implement the classification method for hand-drawn sketches as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, A classification program for hand-drawn sketches is stored on the storage medium, and when the classification program for hand-drawn sketches is executed by a processor, it implements the classification method for hand-drawn sketches as described in any one of claims 1 to 5.

Images