A patent drawing processing method and device and electronic equipment

Abstract

The application provides a patent drawing processing method and device and electronic equipment, and relates to the technical field of patent retrieval. The method is applied to a server and includes: acquiring a patent drawing with a drawing mark; splitting and linearly mapping the patent drawing to obtain a patent drawing layer; inputting the patent drawing layer into a Transformer model to obtain a feature pyramid, wherein the feature pyramid includes semantic features and spatial features of the drawing mark, the semantic features are text content of the drawing mark, and the spatial features are positions of the drawing mark in the patent drawing; inputting the feature pyramid into an FPEM stacked model to output an enhanced feature map, wherein the enhanced feature map includes the semantic features and the spatial features of the drawing mark; performing feature recognition on the enhanced feature map, performing feature processing on the enhanced feature map, and obtaining a text position of the drawing mark. The application has the effect of accurately extracting features of the drawing mark.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of patent retrieval, and in particular to a patent figure processing method and device and electronic equipment. BACKGROUND

[0002] With the development of technology, the field of patent retrieval is becoming more and more mature. Intelligent figures refer to automatically displaying the names of technical features corresponding to figure markers in the vicinity of the figure markers of patent figures, so as to facilitate readers to quickly understand the names of technical features corresponding to figure markers.

[0003] Currently, in intelligent figures, figure markers in the figures need to be extracted and recognized. In the existing extraction method, for some figures with low pixels, since the patent figures themselves have low pixels, the extracted features are relatively blurred, so it is difficult to accurately extract the features of the figure markers, and therefore there is a problem of being difficult to accurately position the figure markers.

[0004] Therefore, in view of the above problem of being difficult to accurately position the figure markers, there is an urgent need for a patent figure processing method, device and electronic equipment. SUMMARY

[0005] The present application provides a patent figure processing method, device and electronic equipment, which has the effect of facilitating accurate positioning of figure markers.

[0006] In a first aspect of the present application, a patent figure processing method is provided, applied to a server, the method comprising:

[0007] obtaining a patent figure with figure markers;

[0008] splitting and linearly mapping the patent figure to obtain a patent figure layer;

[0009] inputting the patent figure layer into a Transformer model to obtain a feature pyramid, wherein the feature pyramid includes semantic features and spatial features of the figure markers, the semantic features are text content of the figure markers, and the spatial features are positions of the figure markers in the patent figure;

[0010] inputting the feature pyramid into an FPEM stacking model to output an enhanced feature map, the enhanced feature map including semantic features and spatial features of the figure markers;

[0011] performing feature processing on the enhanced feature map to obtain a text position of the figure markers.

[0012] By adopting the technical solution, the patent drawing is first split and linearly mapped to obtain a patent drawing layer, and then the patent drawing layer is input into a Transformer model to obtain a feature pyramid. Since the feature pyramid includes semantic features and spatial features of the drawing mark, the semantic features and spatial features of the drawing mark can be extracted into different levels for targeted storage, thereby improving the accuracy of extracting the features of the drawing mark.

[0013] The FPEM stacking model is introduced to facilitate feature fusion of the stored semantic features and spatial features, so as to obtain an enhanced feature map after the semantic features and spatial features are fused. After the server performs feature recognition on the enhanced feature map, the text position of the drawing mark is obtained, thereby achieving accurate positioning of the drawing mark.

[0014] Optionally, the splitting and linear mapping of the patent drawing to obtain a patent drawing layer comprises:

[0015] The patent drawing is split into a plurality of patent sub-graph blocks.

[0016] The plurality of patent sub-graph blocks are straightened to obtain the patent drawing layer.

[0017] By adopting the technical solution, the patent drawing is split into a plurality of patent sub-graph blocks, which facilitates the extraction of the drawing mark in the patent drawing. Through the straightening process, the information of the patent sub-graph block containing the drawing mark is amplified, thereby improving the accuracy of the extracted features of the patent drawing.

[0018] Optionally, the Transformer model includes a first-stage Transformer block, a second-stage Transformer block, a third-stage Transformer block, and a fourth-stage Transformer block, and the first-stage Transformer block, the second-stage Transformer block, the third-stage Transformer block, and the fourth-stage Transformer block are connected in series.

[0019] By adopting the technical solution, the first-stage Transformer block, the second-stage Transformer block, the third-stage Transformer block, and the fourth-stage Transformer block are configured to extract the semantic features and spatial features of all drawing marks in the patent drawing, thereby reducing the probability of missing the semantic features and spatial features of the drawing mark.

[0020] Optionally, the patent drawing layer output by the first-stage or second-stage Transformer block is subjected to transposed convolution and up-sampling processing to facilitate extraction of the spatial features of the drawing mark.

[0021] Optionally, the patent drawing layer output by the fourth-stage Transformer block is subjected to pooling and down-sampling processing to facilitate extraction of the semantic features of the drawing mark.

[0022] Optionally, the feature pyramid output by the FPEM stacking module is subjected to up-sampling and stitching processing to form an enhanced feature map.

[0023] By adopting the above technical solutions, the feature pyramid can be converted into an enhanced feature map after the semantic features and spatial features are fused.

[0024] In a second aspect of the present application, a patent drawing processing device is provided, the device being a server, and the device comprising an acquisition module, a segmentation module, a conversion module, a feature fusion module, and a processing module.

[0025] The acquisition module is configured to acquire a patent drawing with a drawing mark.

[0026] The segmentation module is configured to split and linearly map the patent drawing to obtain a patent drawing layer.

[0027] The conversion module is configured to input the patent drawing layer into a Transformer model to obtain a feature pyramid, wherein the feature pyramid comprises semantic features and spatial features of the drawing mark, the semantic features being text content of the drawing mark, and the spatial features being a position of the drawing mark in the patent drawing.

[0028] The feature fusion module is configured to input the feature pyramid into an FPEM stacking model to output an enhanced feature map, wherein the enhanced feature map comprises the semantic features and spatial features of the drawing mark.

[0029] The processing module is configured to perform feature processing on the enhanced feature map to obtain a text position of the drawing mark.

[0030] Optionally, the segmentation module splits and linearly maps the patent drawing to obtain a patent drawing layer, and specifically comprises:

[0031] The segmentation module is configured to split the patent drawing into a plurality of patent sub-graph blocks, and straighten the plurality of patent sub-graph blocks to obtain the patent drawing layer.

[0032] In a third aspect of the present application, an electronic device is provided, comprising a processor, a memory, a user interface and a network interface, the memory is configured to store instructions, the user interface and the network interface are configured to communicate with other devices, and the processor is configured to execute the instructions stored in the memory to enable the electronic device to perform the method according to any one of the preceding aspects.

[0033] In summary, the present application includes at least one of the following beneficial technical effects:

[0034] 1. The present application first splits and linearly maps the patent drawings to obtain a patent drawing layer, and then inputs the patent drawing layer into a Transformer model to obtain a feature pyramid. Since the feature pyramid includes semantic features and spatial features of the drawing marks, the semantic features and spatial features of the drawing marks can be extracted into different layers for targeted storage, thereby improving the accuracy of extracting features of the drawing marks.

[0035] 2. In the method of the present application, a FPEM stacking model is introduced to facilitate feature fusion of the separately stored semantic features and spatial features, thereby facilitating the obtaining of enhanced feature maps after the fusion of semantic features and spatial features. After processing the enhanced feature maps by the server, the text position of the drawing marks can be detected. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a flowchart of a patent drawing processing method disclosed by an embodiment of the present application.

[0037] Figure 2 is a framework diagram of a patent drawing processing method disclosed by an embodiment of the present application.

[0038] Figure 3 is a schematic diagram of splitting a patent drawing disclosed by an embodiment of the present application.

[0039] Figure 4 is a structural diagram of a patent drawing processing device disclosed by an embodiment of the present application.

[0040] Figure 5 is a structural diagram of an electronic device disclosed by an embodiment of the present application.

[0041] Explanation of reference numerals: 41, acquisition module; 42, segmentation module; 43, conversion module; 44, feature fusion module; 45, processing module; 500, electronic device; 501, processor; 502, communication bus; 503, user interface; 504, network interface; 505, memory. DETAILED DESCRIPTION

[0042] In order for those skilled in the art to better understand the technical solutions in the specification, the technical solutions in the specification will be clearly and completely described below in conjunction with the drawings in the specification. Obviously, the described embodiments are only some of the embodiments of the present application, not all.

[0043] In the description of the embodiments of the present application, the words such as "for example" or "for instance" are used to represent an example, illustration or description. Any embodiment or design scheme described as "for example" or "for instance" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words such as "for example" or "for instance" are intended to present the relevant concept in a specific manner.

[0044] In the description of the embodiments of the present application, the term "a plurality of" means two or more. For example, a plurality of systems means two or more systems, and a plurality of screen terminals means two or more screen terminals. In addition, the terms "first" and "second" are used only for the purpose of description, and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features. The terms "include", "contain", "have" and their variants mean "include but are not limited to", unless otherwise specifically emphasized.

[0045] Before introducing the embodiments of the present application, some terms involved in the embodiments of the present application are defined and explained.

[0046] Transformer model: a kind of neural network that learns context by tracking relationships in sequential data and learns the meaning contained in the context. It applies a set of evolving mathematical techniques called attention or self-attention to detect the way data elements in a sequence influence and depend on each other.

[0047] Convolution operation: convolution operation refers to starting from the top left corner of the image, opening an active window with the same size as the template, multiplying the window image corresponding to the template pixel and adding it, and replacing the brightness value of the center pixel of the window with the calculation result. Then, the active window moves one column to the right and performs the same operation. In this way, from left to right and from top to bottom, a new image can be obtained.

[0048] Transposed Convolution: It is commonly used in semantic segmentation or GAN, and its main function is to do up sampling. In some places, transposed convolution is also called fractionally-strided convolution. The operation steps of transposed convolution can be summarized as follows: 1. Fill s-1 rows and 0 columns between input enhanced feature map elements (where s represents the stride of transposed convolution); 2. Fill k-p-1 rows and 0 columns between input enhanced feature Figure Four maps (where k represents the kernel size of transposed convolution, and p is the padding of transposed convolution, which is different from the padding in convolution operation); 3. Flip the convolution kernel parameters up and down, left and right; 4. Do normal convolution operation (fill 0, stride 1).

[0049] Pooling: It is a method of compressing pictures. Because the larger the picture, the slower the computer processing speed and the more difficult the computer recognition, through pooling processing, it is convenient to reduce the size of the picture.

[0050] Up sampling: It actually means enlarging the image, which refers to the technology that can make the image become higher resolution. In the convolutional neural network, because the size of the output is often smaller after the input image is extracted by the convolutional neural network (CNN), sometimes it is necessary to restore the image to the original size for further calculation (such as image semantic segmentation). This operation of mapping the image from small resolution to large resolution is called up sampling. There are generally three ways to realize up sampling: 1. Interpolation, which generally uses bilinear interpolation because it has the best effect. Although it is more complex in calculation than other interpolation methods, it is negligible compared with convolution calculation. Other interpolation methods include nearest neighbor interpolation and trilinear interpolation. 2. Transposed convolution or deconvolution, which fills 0 between the input feature map and then performs standard convolution calculation, so that the size of the output feature map is larger than the input. 3. Up-Pooling Max Unpooling && AvgUnpooling Max Unpooling, which records the index position of the maximum value in the symmetric maxpooling position, and then places the corresponding value to the original maximum value position in the unpooling stage, and fills 0 in the remaining positions.

[0051] Down-sampling: In fact, it is to reduce the image, the main purpose is to make the image meet the size of the display area, generate a thumbnail of the corresponding image. For example, the pooling layer or convolution layer in the middle is down-sampling. However, the image becomes smaller due to the convolution process is to extract features, and the pooling down-sampling is to reduce the dimension of the feature. Down-sampling mainly adopts two ways: 1, using the pooling layer with stride 2, such as Max-pooling and Average-pooling, and Max-pooling is usually used at present because it is simple to calculate and can better preserve the texture features. 2, using the convolution layer with stride 2, the process of down-sampling is a process of information loss, and the pooling layer is not learnable. Using the learnable convolution layer with stride 2 to replace the pooling layer can get better results, of course, at the same time, it also increases a certain amount of calculation.

[0052] FPEM stacked model: It can be regarded as a lightweight enhanced feature pyramid network (FPN), and the calculation amount is small, and it can be cascaded to achieve the effect of continuously enhancing the features.

[0053] The technical scheme provided by the application can be applied to the reading scene of patent documents. The patent drawing is a patent specification drawing or an abstract drawing, and the text included in the patent drawing at least includes a drawing mark. In the patent retrieval scene, the technical scheme provided by the application can accurately identify the drawing mark in the patent drawing, accurately identify the semantic features and spatial features of the drawing mark, so as to facilitate the subsequent detection task of the text. At present, the CNN model is usually used to identify the drawing mark of the patent drawing, but the application creatively introduces the Transformer model to extract the features of the patent drawing. Because the Transformer improves the slow training defect of the CNN, the self-attention mechanism is used to realize fast parallel. And the Transformer can be increased to a very deep depth, fully excavates the characteristics of the DNN model, and greatly improves the model accuracy.

[0054] The application provides a patent drawing processing method, referring to Figure 1 , Figure 1 is a flowchart of a patent drawing processing method disclosed by an embodiment of the application. The patent drawing processing method is applied to a server and includes steps S110 to S130, and the above steps are as follows:

[0055] S110, obtaining a patent drawing with a drawing mark.

[0056] S120, splitting and linear mapping the patent drawing to obtain a patent drawing layer.

[0057] In a possible implementation, the patent drawing is split and linearly mapped to obtain a patent drawing layer, including: splitting the patent drawing into a plurality of patent sub-patches; and performing straightening processing on the plurality of patent sub-patches to obtain the patent drawing layer.

[0058] Referring to Figure 2 and Figure 3 , Figure 2 is a schematic diagram of a patent drawing processing method according to an embodiment of the present application. Figure 3 is a schematic diagram of splitting a patent drawing according to an embodiment of the present application.

[0059] For example, the pixels of a patent drawing are 1200*1200, and the server splits the patent drawing into a plurality of patent sub-patches of the same size, and the pixels of each patent sub-patch are 100*100*3 in three dimensions. Then, straightening processing is performed on each patent sub-patch, for example, each patent sub-patch is straightened to one dimension of 30000 pixels, thereby combining to form a patent drawing layer.

[0060] Specifically, referring to Figure 3 , the reference numeral "301" and the reference numeral "302" are in different patent sub-patches after the patent drawing is split at a constant ratio.

[0061] In S130, the patent drawing layer is input into a Transformer model to obtain a feature pyramid, wherein the feature pyramid includes semantic features and spatial features of the reference numerals, the semantic features are text contents of the reference numerals, and the spatial features are positions of the reference numerals in the patent drawing. Specifically, referring to Figure 3 , the semantic features are the reference numeral "301" and the reference numeral "302", and the spatial features are coordinates of the reference numeral "301" and the reference numeral "302" in the patent drawing.

[0062] In S140, the feature pyramid is input into an FPEM stacking model to output an enhanced feature map, and the enhanced feature map includes the semantic features and the spatial features of the reference numerals.

[0063] In S150, feature processing is performed on the enhanced feature map to obtain text positions of the reference numerals.

[0064] In a possible implementation, the Transformer model comprises a first-stage Transformer block, a second-stage Transformer block, a third-stage Transformer block, and a fourth-stage Transformer block, and the first-stage Transformer block, the second-stage Transformer block, the third-stage Transformer block, and the fourth-stage Transformer block are connected in series.

[0065] In a possible implementation, the patent drawing layer output by the first-stage Transformer block or the second-stage Transformer block is subjected to transposed convolution and up-sampling processing, so as to extract spatial features of the drawing mark.

[0066] In a possible implementation, the patent drawing layer output by the third-stage Transformer block can further extract semantic features of the drawing mark.

[0067] In a possible implementation, the patent drawing layer output by the fourth-stage Transformer block is subjected to pooling and down-sampling processing, so as to extract semantic features of the drawing mark.

[0068] In a possible implementation, with reference to Figure 2 , the patent drawing layer sequentially passes through the first-stage Transformer block, the second-stage Transformer block, the third-stage Transformer block, and the fourth-stage Transformer block, so as to gradually and finely extract semantic features of the drawing mark, thereby highlighting the semantic features and spatial features of the drawing mark.

[0069] Specifically, the patent drawing is sequentially subjected to the first-stage Transformer block, the second-stage Transformer block, the third-stage Transformer block, and the fourth-stage Transformer block, and then subjected to 4 times upsampling by 2*2 transposed convolution with a step of 2 to form the first feature pyramid block, subjected to 2 times upsampling by 2*2 transposed convolution with a step of 1 after the second-stage Transformer block to form the third feature pyramid block, subjected to 2 times downsampling by 2*2 pooling processing with a step of 1 after the fourth-stage Transformer block to form the second feature pyramid block, and subjected to 4 times upsampling by 2*2 transposed convolution with a step of 2 after the third-stage Transformer block to form the fourth feature pyramid block. Since the patent drawing is sequentially subjected to the first-stage Transformer block, the second-stage Transformer block, the third-stage Transformer block, and the fourth-stage Transformer block, the first feature pyramid block, the second feature pyramid block, the third feature pyramid block, and the fourth feature pyramid block are sequentially arranged from bottom to top, thereby forming the feature pyramid.

[0070] Specifically, referring to Figure 2 , the patent sub-drawing block is sequentially subjected to the first-stage Transformer block, the second-stage Transformer block, the third-stage Transformer block, and the fourth-stage Transformer block, thereby facilitating extraction of more spatial features and semantic features of the patent drawing marks. The spatial features and semantic features of the blank positions in the patent drawing are sequentially removed, thereby facilitating the extracted spatial features and semantic features of the patent drawing marks to be sufficient, reducing the probability of missing the patent drawing marks, and making the extracted positions of the patent drawing marks more accurate.

[0071] In a possible implementation, the patent drawing processing method further includes upsampling and splicing the feature pyramid output by the FPEM stacking module to form an enhanced feature map.

[0072] Since the low-layer feature semantic information is less but the target position is accurate, and the high-layer feature semantic information is rich but the target position is rough, the image is subjected to multi-scale enhanced feature map extraction at different levels, and then the enhanced feature maps are fused, so as to combine the advantages of low-layer and high-layer features. The feature pyramid network is generated based on such an idea.

[0073] In the FPEM stacking module, a set of input multi-scale feature pyramid blocks (for example, each feature pyramid block of different scales output by the image through the feature pyramid network) are processed by different levels of convolution layers to generate feature maps of different scales, and are output to different levels. The enhanced feature maps of deeper levels are up-sampled to the same size as the enhanced feature maps of shallow levels and are stacked, and the process is repeated to form the up-sampling strengthening process in the upper half of the figure, thereby forming the up-sampling strengthened enhanced feature maps. Then, the up-sampling strengthened enhanced feature maps are subjected to the down-sampling strengthening process in the lower half of the figure, that is, the up-sampling strengthened enhanced feature maps are output and fused with the enhanced feature maps of the corresponding level in the down-sampling process, and are transmitted to the next level. In this way, a set of enhanced feature maps with the same size as the input set of enhanced feature maps are finally output.

[0074] Specifically, referring to Figure 2 , after the feature pyramid passes through the FPEM stacking module, the feature pyramid is subjected to up-sampling and splicing processing, the semantic features and spatial features of the reference numerals included in the feature pyramid are fused, so that the extracted information only includes the semantic features and spatial features of the reference numerals, the information of the blank positions in the reference numerals is eliminated, and the extracted information of the reference numerals is more accurate.

[0075] The application also provides a patent drawing processing device, which is a server. Referring to Figure 4 , Figure 4 is a structural schematic diagram of a patent drawing processing device disclosed in an embodiment of the application. The device comprises an acquisition module 41, a segmentation module 42, a conversion module 43, a feature fusion module 44, and a processing module 45.

[0076] The acquisition module 41 is configured to acquire a patent drawing with reference numerals.

[0077] The segmentation module 42 is configured to split and linearly map the patent drawing to obtain a patent drawing layer.

[0078] The conversion module 43 is configured to input the patent drawing layer into a Transformer model to obtain a feature pyramid, wherein the feature pyramid comprises semantic features and spatial features of the reference numerals, the semantic features are text contents of the reference numerals, and the spatial features are positions of the reference numerals in the patent drawing.

[0079] The feature fusion module 44 is configured to input the feature pyramid into an FPEM stacking model to output enhanced feature maps, wherein the enhanced feature maps comprise the semantic features and the spatial features of the reference numerals.

[0080] The processing module 45 is configured to process the enhanced feature maps to obtain text positions of the reference numerals.

[0081] In a possible implementation, the splitting module 42 splits and linearly maps the patent drawings to obtain patent drawing layers, specifically including:

[0082] The splitting module 42 is configured to split the patent drawings into a plurality of patent sub-patches, and perform straightening processing on the plurality of patent sub-patches to obtain the patent drawing layers.

[0083] In a possible implementation, the Transformer model includes a first-stage Transformer block, a second-stage Transformer block, a third-stage Transformer block, and a fourth-stage Transformer block, and the first-stage Transformer block, the second-stage Transformer block, the third-stage Transformer block, and the fourth-stage Transformer block are connected in series.

[0084] In a possible implementation, the patent drawing layers output by the first-stage Transformer block or the second-stage Transformer block are subjected to transposed convolution and up-sampling processing, so as to extract spatial features of the drawing marks.

[0085] In a possible implementation, the patent drawing layers output by the fourth-stage Transformer block are subjected to pooling and down-sampling processing, so as to extract semantic features of the drawing marks.

[0086] In a possible implementation, the feature fusion module is further configured to perform up-sampling and splicing processing on the feature pyramid output by the FPEM stacking module to form an enhanced feature map.

[0087] It should be noted that, when the apparatuses provided in the above embodiments implement their functions, the above-described division of the functional modules is only used as an example for description, and in actual application, the above-described functions can be completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the apparatuses and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be described here.

[0088] The present application also discloses an electronic device. Referring to Figure 5 , Figure 5 is a structural schematic diagram of an electronic device disclosed by the embodiments of the present application. The electronic device 500 can include at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0089] The communication bus 502 is configured to realize the connection communication between the components.

[0090] The user interface 503 can include a display and a camera. Optionally, the user interface 503 can further include a standard wired interface and a wireless interface.

[0091] The network interface 504 can include a standard wired interface and a wireless interface (e.g., a WI-FI interface).

[0092] The processor 501 can include one or more processing cores. The processor 501 is connected to various parts of the server through various interfaces and lines, and performs various functions and processes data of the server by running or executing instructions, programs, code sets or instruction sets stored in the memory 505, and calling data stored in the memory 505. Optionally, the processor 501 can be implemented in at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 501 can be integrated with a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU is mainly used to process an operating system, a user interface, and an application program. The GPU is used to render and draw the content to be displayed on the display. The modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 501, but can be implemented by a separate chip.

[0093] The memory 505 may include random access memory (RAM) or read-only memory. Optionally, the memory 505 may include a non-transitory computer-readable storage medium. The memory 505 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 505 may also be at least one storage device located remotely from the aforementioned processor 501. (Refer to...) Figure 5 The memory 505, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for processing patent drawings.

[0094] exist Figure 5 In the illustrated electronic device 500, the user interface 503 is mainly used to provide an input interface for the user and to acquire user input data; while the processor 501 can be used to call an application program stored in the memory 505 containing a processing method of a patent drawing. When executed by one or more processors 501, the electronic device 500 performs one or more of the methods described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0095] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0096] In several embodiments provided in the present application, it should be understood that the disclosed apparatus can be implemented in other manners. For example, the division of the apparatus embodiments is merely illustrative, and the division of units can be changed according to actual conditions, such as a combination or integration of some units, or a deletion of some features, or an addition of some features. In addition, the coupling or direct coupling or communication connection between the shown or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0097] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0098] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0099] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable memory. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned memory includes: a U disk, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0100] The above is only exemplary embodiments of the present disclosure, which cannot limit the scope of the present disclosure. Any equivalent changes and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Those skilled in the art will easily think of other embodiments of the present disclosure after considering the specification and the true disclosure.

[0101] The present application is intended to cover any variations, uses or adaptive changes of the present disclosure that follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not described in the present disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method of processing patent drawings, characterized by, The method applied to a server comprises: S110, acquiring a patent drawing with a drawing mark; S120, splitting and linearly mapping the patent drawing to obtain a patent drawing layer; Step S120 includes: cutting the patent drawing into a plurality of patent sub-patches; The plurality of patent sub-patches are straightened to obtain the patent drawing layer; S130, inputting the patent drawing layer into a Transformer model to obtain a feature pyramid, wherein the feature pyramid comprises semantic features and spatial features of the drawing mark, the semantic features are text content of the drawing mark, and the spatial features are positions of the drawing mark in the patent drawing; The Transformer model comprises a first-stage Transformer block, a second-stage Transformer block, a third-stage Transformer block, and a fourth-stage Transformer block, and the first-stage Transformer block, the second-stage Transformer block, the third-stage Transformer block, and the fourth-stage Transformer block are connected in series; The patent drawing layer output by the first-stage Transformer block or the second-stage Transformer block is processed by transposed convolution and up-sampling to extract the spatial features of the drawing mark, and after the first-stage Transformer block, 4 times up-sampling is performed to form a first feature pyramid block, and after the second-stage Transformer block, 2 times up-sampling is performed to form a third feature pyramid block; The patent drawing layer output by the fourth-stage Transformer block is processed by pooling and down-sampling to extract the semantic features of the drawing mark, and after the fourth-stage Transformer block, 2 times down-sampling is performed to form a second feature pyramid, and after the third-stage Transformer block, a fourth feature pyramid block is formed, and the first feature pyramid block, the second feature pyramid block, the third feature pyramid block, and the fourth feature pyramid block are arranged from bottom to top in turn to form a feature pyramid; S140, inputting the feature pyramid into an FPEM stacking model to output an enhanced feature map, wherein the enhanced feature map comprises the semantic features and the spatial features of the drawing mark; S150, processing the enhanced feature map to obtain a text position of the drawing mark.

2. The method of processing patent drawings according to claim 1, wherein, The feature pyramid output by the FPEM stacking model is up-sampled and spliced to form an enhanced feature map.

3. A patent drawing processing apparatus characterized by comprising: The device is a server, and the device comprises an acquisition module (41), a segmentation module (42), a conversion module (43), a feature fusion module (44), and a processing module (45); The acquisition module (41) is configured to acquire a patent drawing with a drawing mark; The segmentation module (42) is configured to split and linearly map the patent drawing to obtain a patent drawing layer; The segmentation module (42) is configured to segment the patent drawing into a plurality of patent sub-patches, and to perform straightening processing on the plurality of patent sub-patches to obtain the patent drawing layer; The conversion module (43) is configured to input the patent drawing layer into a Transformer model to obtain a feature pyramid, wherein the feature pyramid includes semantic features and spatial features of the drawing mark, the semantic features are text content of the drawing mark, and the spatial features are positions of the drawing mark in the patent drawing; The Transformer model includes a first-stage Transformer block, a second-stage Transformer block, a third-stage Transformer block, and a fourth-stage Transformer block, and the first-stage Transformer block, the second-stage Transformer block, the third-stage Transformer block, and the fourth-stage Transformer block are connected in series; The patent drawing layer output by the first-stage Transformer block or the second-stage Transformer block is subjected to transpose convolution and up-sampling processing to extract the spatial features of the drawing mark, is subjected to 4 times up-sampling after the first-stage Transformer block to form a first feature pyramid block, and is subjected to 2 times up-sampling after the second-stage Transformer block to form a third feature pyramid block; The patent drawing layer output by the fourth-stage Transformer block is subjected to pooling and down-sampling processing to extract the semantic features of the drawing mark, is subjected to 2 times down-sampling after the fourth-stage Transformer block to form a second feature pyramid, is subjected to down-sampling after the third-stage Transformer block to form a fourth feature pyramid block, and the first feature pyramid block, the second feature pyramid block, the third feature pyramid block, and the fourth feature pyramid block are arranged in order from bottom to top to form the feature pyramid; The feature fusion module (44) is configured to input the feature pyramid into an FPEM stacking model to output an enhanced feature map, and the enhanced feature map includes the semantic features and the spatial features of the drawing mark; The processing module (45) is configured to perform feature processing on the enhanced feature map to obtain text positions of the drawing mark.

4. An electronic device, comprising: The electronic device (500) includes a processor (501), a memory (505), a user interface (503), and a network interface (504), the memory (505) is configured to store instructions, the user interface (503) and the network interface (504) are configured to communicate with other devices, and the processor (501) is configured to execute the instructions stored in the memory (505) to enable the electronic device (500) to perform the method of any one of claims 1-2.

Images