Copying image recognition method, device, terminal equipment and storage medium

Abstract

The present application belongs to the field of computer vision technology and provides a copy image recognition method, apparatus, terminal device and storage medium. The copy image recognition method includes: extracting a room type feature map and a room boundary feature map from the acquired copy image, determining an attention weight according to the room type feature map, and fusing the room boundary feature map according to the room type feature map and the attention weight to output a classification result of the room boundary. Since the attention weight is learned by learning the room type features and further fused with the room boundary feature map, the features of the room type and the features of the room boundary are associated, that is, since the attention weight has the characteristics of modeling the importance of each feature channel and enhancing or suppressing different channels for different tasks, the detection accuracy of the room boundary is greatly improved.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and in particular relates to a method, apparatus, terminal device and storage medium for recognizing a copied image. Background Art

[0002] In the field of copied image recognition, traditional recognition methods are generally used. For example, graphics methods are used to detect shapes in images, and then infer the types of walls, doors, and rooms based on the shapes; or after filtering out text content, various thick and thin lines are detected, and then the detected thin strips are located as walls, doors, and windows; or other heuristic detection methods are used to extract all contours using straight line detection, and then identify walls, doors, windows, and room types based on color.

[0003] Heuristic detection methods have many flaws, such as interference from numerous ruler lines, irregular wall shapes, and traces of varying styles. These factors severely impact detection accuracy, and thus the overall detection performance. Existing deep learning methods, such as convolutional neural networks (CNNs), can improve detection accuracy, but their classification accuracy for traced image pixels is limited. Simply increasing the number of pixel classifications can lead to a decrease in detection accuracy. This can lead to insufficient correlation between walls, doors, windows, and room types, reducing the accuracy of room boundary detection. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a method, apparatus, terminal device, and storage medium for copying image recognition to improve the detection accuracy of copying image recognition in the prior art.

[0005] A first aspect of an embodiment of the present application provides a method for recognizing a copied image, comprising:

[0006] Extracting a room type feature map and a room boundary feature map from the acquired copy image;

[0007] Determine the attention weight based on the room type feature map;

[0008] According to the room type feature map and the attention weight, the room boundary feature map is fused to output the classification result of the room boundary.

[0009] In some implementations, obtaining an attention weight according to the room type feature map includes:

[0010] Perform feature mapping on the room type feature map to obtain a room sub-feature map;

[0011] The weight of the room sub-feature map is used as the first attention weight;

[0012] The first attention weight is aggregated with the preset direction-aware convolution kernel to obtain the second attention weight.

[0013] In some embodiments, fusing the room boundary feature map according to the room type feature map and the attention weight to output a classification result of the room boundary includes:

[0014] Obtain the boundary sub-feature map of the room boundary feature map;

[0015] Perform a first aggregation on the boundary sub-feature map and the second attention weight to obtain a first boundary map;

[0016] Performing a second aggregation on the first boundary map and the first attention weight to obtain a second boundary map;

[0017] The room boundary classification result is determined according to the room boundary feature map and the second boundary map.

[0018] In some embodiments, determining a room boundary classification result based on the room boundary feature map and the second boundary map includes:

[0019] Concatenate the room boundary feature map and the second boundary map to obtain a boundary classification map;

[0020] Input the boundary classification map into the preset classification network and output the classification probability corresponding to the boundary classification map;

[0021] The boundary result corresponding to the maximum classification probability is determined as the room boundary classification result.

[0022] In some embodiments, extracting a room type feature map and a room boundary feature map from the acquired copy image includes:

[0023] Perform convolution pooling on the copied image to obtain the first shared feature map;

[0024] The first shared feature map is subjected to a dilated convolution process to obtain a second shared feature map;

[0025] The second shared feature map is decoded to output a room type feature map and a room boundary feature map.

[0026] In some embodiments, extracting a room type feature map and a room boundary feature map from the acquired copy image includes:

[0027] Performing convolution pooling processing on the copied image to obtain a first shared feature map;

[0028] Extracting a second shared feature map from the first shared feature map through a preset residual network;

[0029] The second shared feature map is decoded to output a room type feature map and a room boundary feature map.

[0030] A second aspect of an embodiment of the present application provides a copy image recognition device, comprising:

[0031] An acquisition module, configured to extract a room type feature map and a room boundary feature map from the acquired copy image;

[0032] Attention module, which is used to determine attention weights based on the room type feature map;

[0033] The classification module is used to fuse the room boundary feature map according to the room type feature map and the attention weight to output the classification result of the room boundary.

[0034] In some implementations, the attention module includes:

[0035] A mapping unit, configured to perform feature mapping on the room type feature map to obtain a room sub-feature map;

[0036] The first attention unit is used to take the weight of the room sub-feature map as the first attention weight;

[0037] The second attention unit is used to aggregate the first attention weight and the preset direction-aware convolution kernel to obtain the second attention weight.

[0038] In some embodiments, a classification module comprises:

[0039] An acquisition unit, configured to acquire a boundary sub-feature map of a room boundary feature map;

[0040] A first aggregation unit is configured to perform a first aggregation on the boundary sub-feature map and the second attention weight to obtain a first boundary map;

[0041] a second aggregation unit, configured to perform a second aggregation on the first boundary map and the first attention weight to obtain a second boundary map;

[0042] The classification unit is used to determine a room boundary classification result according to the room boundary feature map and the second boundary map.

[0043] In some embodiments, a taxonomic unit comprises:

[0044] A concatenation subunit, configured to concatenate the room boundary feature map and the second boundary map to obtain a boundary classification map;

[0045] The classification probability subunit is used to input the boundary classification map into the preset classification network and output the classification probability corresponding to the boundary classification map;

[0046] The determination subunit is used to determine the boundary result corresponding to the maximum classification probability as the room boundary classification result.

[0047] In some embodiments, the acquisition module includes:

[0048] A convolution and pooling unit, configured to perform convolution and pooling processing on the copied image to obtain a first shared feature map;

[0049] a dilated convolution unit, configured to perform dilated convolution on the first shared feature map to obtain a second shared feature map;

[0050] A decoding unit is configured to decode the second shared feature map to output a room type feature map and a room boundary feature map.

[0051] In some embodiments, the acquisition module includes:

[0052] A convolution and pooling unit, configured to perform convolution and pooling processing on the copied image to obtain a first shared feature map;

[0053] A residual unit, configured to extract a second shared feature map from the first shared feature map through a preset residual network;

[0054] The decoding unit is further configured to decode the second shared feature map to output a room type feature map and a room boundary feature map.

[0055] A third aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned copy image recognition method are implemented.

[0056] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned copy image recognition method are implemented.

[0057] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0058] In the present application, the room type feature map and the room boundary feature map are extracted from the acquired copy image, the attention weight is determined according to the room type feature map, and the room boundary feature map is fused according to the room type feature map and the attention weight to output the classification result of the room boundary. Since the attention weight is learned by the room type feature and further fused with the room boundary feature map, the features of the room type and the features of the room boundary are associated, that is, since the attention weight has the characteristics of modeling the importance of each feature channel and enhancing or suppressing different channels for different tasks, the detection accuracy of the room boundary is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0060] Figure 1 Schematic diagram of the overall network structure of the copy image recognition method according to an embodiment of the present application;

[0061] Figure 2 This is a flow chart of an embodiment of the copy image recognition method of the present application embodiment;

[0062] Figure 3 1 is a flow chart of obtaining a first shared feature map in a copy image recognition method according to an embodiment of the present application;

[0063] Figure 4 Schematic diagram of the structure of the basic modules of the copy image recognition method according to the embodiment of the present application;

[0064] Figure 5 1 is a schematic diagram of a process for obtaining a second shared feature map in a copy image recognition method according to an embodiment of the present application;

[0065] Figure 6 Schematic diagram of the attention structure of the copy image recognition method according to an embodiment of the present application;

[0066] Figure 7 Schematic diagram of the structure of the copy image recognition device provided in an embodiment of the present application;

[0067] Figure 8 This is a schematic diagram of the hardware structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0069] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0070] In the embodiments of the present application, the execution subject of the process is a terminal device, which includes but is not limited to laptops, computers, servers, tablet computers, smartphones, and other terminal devices with software development capabilities. In particular, when executing the processes in the implementation of the present application, the terminal device can be used to provide desktop application hosting functions or front-end interface display for native applications, providing an interface assembly framework for desktop applications.

[0071] In the embodiments of this application, Figure 1 The overall network structure diagram of the copy image recognition method of the embodiment of the present application is as follows: the copy image is input into the shared network for feature extraction, and the shared feature map is output, which is decoded by different decoders in the decoding network. The shared network can be a hollow pyramid convolutional network, a residual network (Resnet) or a visual geometry group network (Visual Geometry Group The shared network improves the receptive field of the extracted shared feature map while reducing edge loss. The decoder includes a decoder for the room boundary feature map and a decoder for the room type feature map. The decoder is composed of different numbers of convolution kernels. Since the same shared feature map is used before decoding, the decoded room boundary feature map and the room type feature map are correlated, which is conducive to learning the attention weight value of the interesting information of the room type from the room type feature map. The interesting information is the shape of the room type for walls of different thicknesses, that is, corresponding to different room boundaries. Different weights are learned through the room type feature map to fuse the room boundary feature map, and then the classification result of the room boundary is output. For example, when the room type feature map identifies that the living room and the dining room are connected, the room boundary cannot be a wall, but is more likely to be a door. At this time, the attention weight corresponding to the room type feature map pays more attention to the door, that is, the attention weight is predicted by inferring the corresponding room type, which reduces the interference of irrelevant information and fuses the room boundary feature map to make the output room boundary more accurate.

[0072] Furthermore, if Figure 2 FIG. 1 is a flowchart of the implementation of the copy image recognition method according to an embodiment of the present application, which specifically includes:

[0073] S201: Extract a room type feature map and a room boundary feature map from the acquired copy map.

[0074] The tracing image is a floor plan image drawn by an architect. This image shows the floor plan layout of the apartment, but does not display the specific semantic information corresponding to each space in the apartment. Semantic information includes, but is not limited to, room type, room boundaries, and room boundary thickness. Room types may include, but are not limited to, living rooms, dining rooms, kitchens, bathrooms, and other rooms. Room boundaries may include walls, doors, windows, and closets, etc., which are not limited here. The acquisition method can be, but is not limited to, images uploaded locally by the user or retrieved from a designated server or database.

[0075] Furthermore, feature extraction from the copied image can be performed using feature extraction networks such as ResNet, Visual Geometry Group Network (VGG), and other convolutional neural networks. For example, ResNet 34 / 50 / 101 / 152 (the number represents the number of network layers; a larger number indicates a deeper network layer) and VGG16 (not limited here) are examples of ResNet.

[0076] The feature extraction network is used to extract features from the copied image, that is, to effectively reduce the dimension of the copied image and extract useful information to remove some redundant interference information. At the same time, it also reduces a certain amount of calculation for subsequent recognition processing and improves the accuracy and efficiency of subsequent recognition of room boundaries.

[0077] In some embodiments, extracting the room type feature map and the room boundary feature map from the acquired copy image specifically includes:

[0078] Perform convolution pooling on the copied image to obtain the first shared feature map;

[0079] The first shared feature map is subjected to a dilated convolution process to obtain a second shared feature map;

[0080] The second shared feature map is decoded to output a room type feature map and a room boundary feature map.

[0081] Specifically, the feature processing of the copied image is divided into two parts, namely the extraction of the first shared feature map and the second shared feature map. Among them, the convolution pooling process refers to the convolution calculation of the convolution layer and the downsampling process of the pooling layer, such as maximum pooling, average pooling, global average pooling and global maximum pooling. In addition to the convolution pooling process, the first shared feature map also includes the processing of the batch normalization layer and the activation layer to improve the expression ability of the feature extraction network. Figure 3 This is a flow chart of obtaining the first shared feature map in the copy image recognition method of the embodiment of the present application. Figure 3After the copied image passes through the convolution layer and the pooling layer, it is processed by a preset number of different types of basic blocks to obtain the first shared feature map. The different types of basic blocks include 3 first basic blocks, 4 second basic blocks, 23 third basic blocks, and 3 fourth basic blocks. The structural network layers of different types of basic blocks are the same, but the parameters set for the structural network layers are different. Figure 4 This is a schematic diagram of the structure of the basic modules of the copy image recognition method of the embodiment of the present application. Figure 4 It can be seen that the basic block is composed of several convolutional layers, several batch normalization layers, and several activation layers. In the embodiment of the present application, when the feature map extracted by the previous layer (i.e., the feature map that has not entered the basic block for processing) is input into different types of basic blocks, the convolution feature map needs to be processed by the convolution layer in sequence, and then the convolution feature map is calculated by the batch normalization layer and the activation layer, and the feature map obtained by the calculation is re-input into the convolution layer, batch normalization layer, and activation layer for a second calculation process, and the third operation is continued on the feature map obtained by the second calculation. At this time, the operation is only processed by the convolution layer and batch normalization layer, and the feature map obtained at this time and the feature map that has not entered the basic block for processing are superimposed, and the superimposed feature map is calculated by the activation layer, and the calculation result of the basic block is output, which means that the execution of the basic block is completed at this time. The combination of convolutional layers, batch normalization layers, and activation layers in the basic block is beneficial for preventing gradient vanishing, controlling gradient explosion, and preventing overfitting during the feature extraction process of the copy image. This ensures that the first shared feature map extracted has more useful information of interest, providing an effective data basis for the subsequent room boundary recognition accuracy.

[0082] Continue to refer Figure 5 , Figure 5 It is a flow chart of obtaining the second shared feature map in the copy image recognition method of the embodiment of the present application. The first shared feature map is input into the dilated convolutional network structure for convolution processing to obtain 3 feature maps, and the feature map is subjected to 1×1 convolution processing to obtain the second shared feature map. In the embodiment of the present application, the dilated convolutional network includes four convolution layers and pooling layers, wherein the convolution layer includes a normal convolution layer with a convolution kernel of 1×1, a dilated convolution layer with a convolution kernel of 3×3 and a dilated convolution rate of 6, a dilated convolution layer with a convolution kernel of 3×3 and a dilated convolution rate of 12, and a dilated convolution layer with a convolution kernel of 3×3 and a dilated convolution rate of 18, wherein the dilation rate refers to the number of intervals of the convolution kernel, and the dilation convolution rate can increase the receptive field in the feature extraction process, so that each convolution output contains a larger range of information, so that the extracted second shared feature map has more effective information of interest, for example, the area, shape and thickness of the boundary of the room type and other information of interest.

[0083] In an embodiment of the present application, the second shared feature map is decoded by different decoders, and different decoders use different convolution kernels. For example, when the decoder used to decode and output the room type feature map is a 64×64×128 convolution kernel, the decoder used to decode and output the room boundary feature map is a 32×32×256 kernel, which is not limited here. Since the second shared feature map is used before decoding, that is, the feature map incorporates features of the same scale, after decoding and output, the obtained room type feature map and room boundary feature map have an associated relationship, that is, the extracted room type feature map and room boundary feature map have the characteristics of spatial context, which is conducive to improving the detection and recognition accuracy.

[0084] In an embodiment of the present application, a convolution pooling process is performed on the copy image to obtain a first shared feature map, the first shared feature map is subjected to a dilated convolution process to obtain a second shared feature map, and the second shared feature map is decoded to output a room type feature map and a room boundary feature map. That is, the multi-scale features of the copy image can be fused through the shared feature map, and there is no need to design feature extraction networks for the room type feature map and the room boundary feature map separately, thereby reducing the time cost and design cost of feature extraction and improving the efficiency of feature extraction. In addition, the same second shared feature map is decoded by different encoders, so that the output room type feature map and the room boundary feature map have an associated relationship, making the extracted feature information more effective and more accurate.

[0085] In the embodiment of the present application, the room type feature map and the room boundary feature map extracted from the acquired copy map may also be:

[0086] Perform convolution pooling on the copied image to obtain the first shared feature map;

[0087] Extracting a second shared feature map from the first shared feature map through a preset residual network;

[0088] The second shared feature map is decoded to output a room type feature map and a room boundary feature map.

[0089] Among them, the convolution pooling process of the first shared feature map here is the same as that described above, and will not be repeated here. The residual network can be ResNet 34 / 50 / 101 / 152. The embodiment of the present application preferably adopts ResNet101, that is, a 101-layer residual network is used to extract the second shared feature map from the first shared feature map. Since the residual network is different from the ordinary network in that it introduces a jump connection, even if the information of the upper layer of the residual network flows into the next layer of the residual network without hindrance, the information flow is improved, and the vanishing gradient problem and degradation problem caused by the network being too deep are avoided. Therefore, Resnet101 has better feature extraction effect than VGG and is helpful for semantic segmentation; in addition, by decoding the same second shared feature map through different encoders, the output room type feature map and the room boundary feature map are correlated, making the extracted feature information more effective and more accurate.

[0090] S202: Determine the attention weight according to the room type feature map.

[0091] The above steps for determining attention weights are specifically as follows:

[0092] Perform feature mapping on the room type feature map to obtain a room sub-feature map;

[0093] The weight of the room sub-feature map is used as the first attention weight;

[0094] The first attention weight is aggregated with the preset direction-aware convolution kernel to obtain the second attention weight.

[0095] Among them, feature mapping refers to reducing the room type feature map to a two-dimensional feature map after a series of convolution calculations, and using the two-dimensional feature map as the first attention weight, wherein the feature mapping refers to the matrix generated after the input room type feature map and the convolution kernel are convolved, that is, the two-dimensional feature map is a room sub-feature map in the form of a matrix. In the embodiment of the present application, the process of feature mapping is as follows Figure 6 In the attention structure diagram shown, a room type feature map is convolved twice with a 3×3 kernel and once with a 1×1 kernel. The activation layer then preserves and maps the features of the sub-feature map, removing some redundant data to produce an N×N×1 sub-feature map. Because the first attention weight is derived from the room type feature map through convolution, which is derived from feature extraction of each input copy, as the copy changes, coupled with convolution learning, the first attention weight becomes more adapted to the interesting information in the copy.

[0096] Furthermore, the preset direction-aware convolution kernels include but are not limited to horizontal convolution kernels, vertical convolution kernels, diagonal convolution kernels, and flipped diagonal convolution kernels. The direction-aware convolution kernel performs weighted summation on the local row and column areas set in the matrix to achieve local perception, such as Figure 6 The four direction-aware convolution kernels shown are four 3×3 matrices, and the shaded part in the matrix is the weight of local perception. For example, the 3×3 horizontal convolution kernel is Figure 6 As shown in the second matrix in [1], the shaded portion of the second row of the second matrix represents the local perception weight of the horizontal convolution kernel. The first attention weight is aggregated with the preset direction-aware convolution kernel. This means multiplying the room sub-feature map with the direction-aware convolution kernel respectively. The resulting four products are then added together to form the second attention weight.

[0097] Since the second attention weight not only learns the field of view of interest from the room sub-feature map, but also combines the local perception of the direction-aware convolution kernel, the value of the obtained second attention weight is more accurate.

[0098] S203: Fusing the room boundary feature map according to the room type feature map and the attention weight to output a classification result of the room boundary.

[0099] The details include:

[0100] Obtain the boundary sub-feature map of the room boundary feature map;

[0101] Perform a first aggregation on the boundary sub-feature map and the second attention weight to obtain a first boundary map;

[0102] Performing a second aggregation on the first boundary map and the first attention weight to obtain a second boundary map;

[0103] The room boundary classification result is determined according to the room boundary feature map and the second boundary map.

[0104] Among them, the room boundary feature map is processed by convolution to obtain the boundary sub-feature map, such as Figure 6 As shown in the figure, after the room boundary feature map undergoes 3×3 convolution and 1×1 convolution, an N×N×1 boundary sub-feature map is output. The first aggregation refers to multiplying the matrix of the boundary sub-feature map with the second attention weight to output the first boundary map, that is, convolving the N×N×1 boundary sub-feature map with the second attention weight, and the resulting convolution result is the N×N×1 first boundary feature map; the second aggregation includes product processing and convolution processing. After multiplying the first boundary feature map with the first attention weight, the convolution processing is performed with a preset convolution kernel to obtain the second boundary map, as shown in the figure. Figure 6The 1×1 convolution kernel shown in FIG; wherein, the boundary sub-feature map, the first boundary map, and the second boundary map are feature maps generated by the room boundary feature map in the calculation of different convolution kernels, and the boundary sub-feature map, the first boundary map, and the second boundary map each represent a different degree of perception of the room boundary.

[0105] In some embodiments, determining a room boundary classification result based on the room boundary feature map and the second boundary map includes:

[0106] Concatenate the room boundary feature map and the second boundary map to obtain a boundary classification map;

[0107] Input the boundary classification map into the preset classification network and output the classification probability corresponding to the boundary classification map;

[0108] The boundary result corresponding to the maximum classification probability is determined as the room boundary classification result.

[0109] Specifically, a boundary classification map is obtained by concatenating the room boundary feature map and the second boundary feature map, and the boundary classification result is determined according to the room classification map. The boundary classification map is a feature map for identifying the corresponding room boundary label. At this time, the channel of the boundary classification map integrates the room boundary feature map and the second boundary feature map, so that the weight values of the pixel points related to the room boundary in the boundary classification map are more prominent, making the detection of the room boundary more accurate.

[0110] The classification network is a pre-trained neural network for classifying room boundaries. It is iteratively trained using a dataset of trace images with labeled room boundary labels. The output of the classification network is the boundary type corresponding to the highest classification probability output by the classification network. Each time, the output of the classification network is compared with the room boundary label carried by the corresponding actual trace image to determine the accuracy of the room boundary classification. When the accuracy falls below a preset threshold, such as 98%, the classification network weights are updated by designing a loss function and a stochastic gradient descent (SGD) training strategy. Training is terminated until the accuracy of room boundary classification reaches the preset threshold, allowing the classification network to classify pixels in the boundary classification image into room boundaries such as doors, windows, and walls.

[0111] Therefore, the boundary classification map obtained above is input into the preset classification network for calculation, and the classification probability corresponding to the boundary classification map is output. The classification probability is the similarity matching between the boundary classification map and each room boundary type. The higher the similarity, the greater the classification probability. The boundary result corresponding to the maximum classification probability is taken as the room boundary classification result.

[0112] In the present application, the room type feature map and the room boundary feature map are extracted from the acquired copy image, the attention weight is determined according to the room type feature map, and the room boundary feature map is fused according to the room type feature map and the attention weight to output the classification result of the room boundary. Since the attention weight is learned by the room type feature and further fused with the room boundary feature map, the features of the room type and the features of the room boundary are associated, that is, since the attention weight has the characteristics of modeling the importance of each feature channel and enhancing or suppressing different channels for different tasks, the detection accuracy of the room boundary is greatly improved.

[0113] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0114] In the embodiment of the present application, a copy image recognition device is also provided, and the copy image recognition device includes modules for performing Figure 3 Each step in the corresponding embodiment. Please refer to Figure 2 Related description in the corresponding embodiment. Figure 7 The schematic diagram of the structure of the copy image recognition device of the present application is shown, including an acquisition module 71, an attention module 72 and a classification module 73:

[0115] An acquisition module 71 is configured to extract a room type feature map and a room boundary feature map from the acquired copy map;

[0116] an attention module 72 for determining an attention weight based on the room type feature map;

[0117] The classification module 73 is used to fuse the room boundary feature map according to the room type feature map and the attention weight to output the classification result of the room boundary.

[0118] In some embodiments, the attention module 72 includes:

[0119] A mapping unit, configured to perform feature mapping on the room type feature map to obtain a room sub-feature map;

[0120] The first attention unit is used to take the weight of the room sub-feature map as the first attention weight;

[0121] The second attention unit is used to aggregate the first attention weight and the preset direction-aware convolution kernel to obtain the second attention weight.

[0122] In some embodiments, the classification module 73 includes:

[0123] An acquisition unit, configured to acquire a boundary sub-feature map of a room boundary feature map;

[0124] A first aggregation unit is configured to perform a first aggregation on the boundary sub-feature map and the second attention weight to obtain a first boundary map;

[0125] a second aggregation unit, configured to perform a second aggregation on the first boundary map and the first attention weight to obtain a second boundary map;

[0126] The classification unit is used to determine a room boundary classification result according to the room boundary feature map and the second boundary map.

[0127] In some embodiments, a taxonomic unit comprises:

[0128] A concatenation subunit, configured to concatenate the room boundary feature map and the second boundary map to obtain a boundary classification map;

[0129] The classification probability subunit is used to input the boundary classification map into the preset classification network and output the classification probability corresponding to the boundary classification map;

[0130] The determination subunit is used to determine the boundary result corresponding to the maximum classification probability as the room boundary classification result.

[0131] In some embodiments, the acquisition module 71 includes:

[0132] A convolution and pooling unit, configured to perform convolution and pooling processing on the copied image to obtain a first shared feature map;

[0133] a dilated convolution unit, configured to perform dilated convolution on the first shared feature map to obtain a second shared feature map;

[0134] A decoding unit is configured to decode the second shared feature map to output a room type feature map and a room boundary feature map.

[0135] In some implementations, the acquisition module 71 further includes:

[0136] A convolution and pooling unit, configured to perform convolution and pooling processing on the copied image to obtain a first shared feature map;

[0137] A residual unit, configured to extract a second shared feature map from the first shared feature map through a preset residual network;

[0138] The decoding unit is further configured to decode the second shared feature map to output a room type feature map and a room boundary feature map.

[0139] Among them, the functional implementation of each module / unit in the above-mentioned copy image recognition device corresponds to the various steps in the above-mentioned copy image recognition method embodiment, and their functions and implementation processes are no longer repeated here.

[0140] Figure 8 This is a schematic diagram of a terminal device provided by an embodiment of the present application. Figure 8 As shown, this embodiment / terminal device 8 includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80, such as a software development program. When the processor 80 executes the computer program 82, the steps in the above-mentioned software development method embodiments are implemented, such as Figure 1 Alternatively, when the processor 80 executes the computer program 82, the functions of the modules / units in the above-mentioned system embodiments are realized, for example, Figure 7 The functions of the acquisition module 71 to the classification module 73 are shown.

[0141] Exemplarily, the computer program 82 can be divided into one or more modules / units, and the one or more modules / units are stored in the memory 81 and executed by the processor 80 to complete the present application. The one or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program 82 in the copy image recognition device / terminal device 8. For example, the computer program 82 can be divided into an acquisition module 71, an attention module 72, and a classification module 73 (modules in the virtual device). The specific functions of each module are as described above and will not be repeated here.

[0142] The terminal device 8 can be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device 8 can include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that Figure 8 It is only an example of the terminal device 8 and does not constitute a limitation on the terminal device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 8 may also include input and output devices, network access devices, buses, etc.

[0143] The processor 80 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0144] The memory 81 may be an internal storage unit of the terminal device 8, such as a hard disk or memory of the terminal device 8. The memory 81 may also be an external storage device of the terminal device 8, such as a plug-in hard disk equipped on the terminal device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Furthermore, the memory 81 may include both an internal storage unit of the terminal device 8 and an external storage device. The memory 81 is used to store the computer program and other programs and data required by the terminal device. The memory 81 may also be used to temporarily store data that has been output or is about to be output.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0146] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0147] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0148] In the embodiments provided in this application, it should be understood that the disclosed systems / terminal devices and methods can be implemented in other ways. For example, the system / terminal device embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection of some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0150] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0151] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0152] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for recognizing a copied image, characterized in that: include: Extracting a room type feature map and a room boundary feature map from the acquired copy image; determining an attention weight according to the room type feature map; fusing the room boundary feature map according to the room type feature map and the attention weight to output a classification result of the room boundary; Acquiring the attention weight according to the room type feature map includes: Performing feature mapping on the room type feature map to obtain a room sub-feature map; The weight of the room sub-feature map is used as the first attention weight; Aggregating the first attention weight and a preset direction-aware convolution kernel to obtain a second attention weight; The fusing the room boundary feature map according to the room type feature map and the attention weight to output a classification result of the room boundary includes: Obtaining a boundary sub-feature map of the room boundary feature map; Performing a first aggregation on the boundary sub-feature map and the second attention weight to obtain a first boundary map; Performing a second aggregation on the first boundary map and the first attention weight to obtain a second boundary map; determining a room boundary classification result according to the room boundary feature map and the second boundary map; Determining a room boundary classification result according to the room boundary feature map and the second boundary map includes: Concatenating the room boundary feature map and the second boundary map to obtain a boundary classification map; Inputting the boundary classification map into a preset classification network and outputting the classification probability corresponding to the boundary classification map; The boundary result corresponding to the maximum classification probability is determined as the room boundary classification result; The step of extracting the room type feature map and the room boundary feature map from the acquired copy map includes: Performing convolution pooling processing on the copied image to obtain a first shared feature map; Performing a dilated convolution on the first shared feature map to obtain a second shared feature map; Decoding the second shared feature map to output a room type feature map and a room boundary feature map; or, The step of extracting the room type feature map and the room boundary feature map from the acquired copy map includes: Performing convolution pooling processing on the copied image to obtain a first shared feature map; Extracting a second shared feature map from the first shared feature map through a preset residual network; The second shared feature map is decoded to output a room type feature map and a room boundary feature map.

2. A copy picture recognition device, characterized in that: include: An acquisition module, configured to extract a room type feature map and a room boundary feature map from the acquired copy image; an attention module, configured to determine an attention weight based on the room type feature map; a classification module, configured to fuse the room boundary feature map according to the room type feature map and the attention weight, so as to output a classification result of the room boundary; The attention module includes: a mapping unit, configured to perform feature mapping on the room type feature map to obtain a room sub-feature map; A first attention unit, configured to use the weight of the room sub-feature map as a first attention weight; a second attention unit, configured to aggregate the first attention weight and a preset direction-aware convolution kernel to obtain a second attention weight; The classification module includes: An acquisition unit, configured to acquire a boundary sub-feature map of a room boundary feature map; A first aggregation unit is configured to perform a first aggregation on the boundary sub-feature map and the second attention weight to obtain a first boundary map; a second aggregation unit, configured to perform a second aggregation on the first boundary map and the first attention weight to obtain a second boundary map; A classification unit, configured to determine a room boundary classification result based on the room boundary feature map and the second boundary map; The classification units include: A concatenation subunit, configured to concatenate the room boundary feature map and the second boundary map to obtain a boundary classification map; The classification probability subunit is used to input the boundary classification map into the preset classification network and output the classification probability corresponding to the boundary classification map; a determination subunit, configured to determine the boundary result corresponding to the maximum classification probability as the room boundary classification result; The acquisition module includes: A convolution and pooling unit, configured to perform convolution and pooling processing on the copied image to obtain a first shared feature map; a dilated convolution unit, configured to perform dilated convolution on the first shared feature map to obtain a second shared feature map; a decoding unit, configured to decode the second shared feature map to output a room type feature map and a room boundary feature map; or, The acquisition module includes: A convolution and pooling unit, configured to perform convolution and pooling processing on the copied image to obtain a first shared feature map; A residual unit, configured to extract a second shared feature map from the first shared feature map through a preset residual network; The decoding unit is further configured to decode the second shared feature map to output a room type feature map and a room boundary feature map.

3. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the copy image recognition method according to claim 1 are implemented.

4. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the copy image recognition method as claimed in claim 1 are implemented.

Images