Equipment facility management method and device based on text detection and recognition model

Abstract

The invention provides an equipment facility validity period automatic management method and device based on text detection and recognition, and the method comprises the following steps: obtaining a first image containing an equipment facility label, and inputting a pre-training text detection model; and outputting a second image of a marked text region through operations such as multi-scale convolution fusion and coding and by combining an edge detection result and through cross attention and edge attention calculation. And identifying the image by using a pre-training text identification model, inputting a result and the image into a key information extraction model, and performing semantic analysis to obtain label information. A security database is constructed to store label information, the validity period of the equipment is judged in real time based on the label information, and a corresponding administrator is notified when the validity period is smaller than a set number of days. According to the scheme, the pre-trained text detection, identification and key information extraction model is utilized to obtain the label information and store the label information in the security database, real-time judgment and notification of the validity period are realized, and the method has the advantages of improving management efficiency, reducing cost, facilitating query and the like.

Description

Technical Field

[0001] The present application relates to the field of equipment and facility management, and in particular to an equipment and facility management method and device based on a text detection and recognition model. Background Art

[0002] Traditional equipment and facilities have always played a fundamental role in ensuring the safety of various places. Common equipment and facilities include fire extinguishers, smoke alarms, and emergency lighting. For a long time, the inspection of the validity period of these equipment and facilities has mainly relied on manual regular inspections and keeping corresponding records. Specifically, staff need to shuttle between various places equipped with equipment and facilities according to the established time period and check each device one by one. For example, in a large shopping mall, staff have to walk through each floor and every corner to check the validity period of fire extinguishers, smoke alarms and other equipment; in an office building, they also need to check each office and corridor to see if the emergency lighting is still within the validity period.

[0003] However, this traditional manual inspection method has many disadvantages. On the one hand, it is extremely time-consuming and labor-intensive, especially for those large-scale and widely distributed places, such as industrial parks and large storage bases. The staff needs to spend a lot of time and energy to complete a complete round of inspection work, which undoubtedly increases the labor cost and management cost.

[0004] On the other hand, and more critically, this method can easily lead to serious consequences due to human negligence. After all, people are not machines, and in long-term repetitive inspection work, it is inevitable that omissions or mistakes will occur. Sometimes, due to momentary carelessness, you may not notice that the expiration date marked on the label of the equipment and facilities has expired, so that the expired equipment cannot be replaced or repaired in time. Once a safety accident such as a fire or emergency power outage occurs, these expired equipment and facilities may not function normally, which will bring huge safety hazards to people's lives and property.

[0005] The limitations of the regular inspection method are becoming more and more prominent in actual applications. When unexpected safety incidents occur suddenly, there is often an embarrassing situation where the equipment and facilities have expired but have not been noticed. In order to prevent such situations where safety incidents cannot be effectively dealt with due to expired equipment, a more conservative approach can only be adopted in many cases, that is, when the equipment and facilities are about to expire, regardless of whether their actual performance can be maintained for a period of time, they are refilled or directly replaced with new equipment. In this way, although it guarantees to a certain extent that the equipment and facilities are available at any time within the validity period, it also causes a great waste of resources, whether it is the resources of the equipment itself, or the human, material, and financial resources consumed for replacement and filling operations.

[0006] To sum up, how to ensure accurate monitoring of the validity period of equipment and facilities while effectively reducing the number of manual inspections, improving management efficiency, and avoiding waste of resources has become a key issue that urgently needs to be solved in the existing technical field. Summary of the invention

[0007] The embodiments of the present application provide an equipment and facility management method and device based on a text detection and recognition model. Through technologies such as multi-scale convolution fusion, edge attention mechanism and multimodal Transformer, automatic detection, recognition and validity period management of equipment and facility label information are realized, thereby improving the automation, precision and intelligence level of equipment and facility management.

[0008] In a first aspect, an embodiment of the present application provides an equipment facility management method based on a text detection and recognition model, the method comprising: Acquire a first image, where the first image is an image containing equipment labels of equipment facilities, input the first image into a pre-trained text detection model to obtain a second image, perform multi-scale convolution fusion on the first image in the text detection model to obtain a multi-scale fusion result, encode the multi-scale fusion result to obtain a text area mask, perform edge detection on the first image to obtain a text edge, perform cross-attention calculation on the text edge and the text area mask after feature multiplication with the first result to obtain a cross result, perform multiple edge attention calculations on the cross result in sequence to obtain multiple edge attention results, perform feature stacking on the cross result and each edge attention result and output to obtain a second image, wherein the first image is referred to as the first result after edge attention calculation, and the second image is the first image marked with a text area; Performing text recognition on the second image using a pre-trained text recognition model to obtain a text recognition result, and then inputting the text recognition result and the second image into a pre-trained key information extraction model, wherein the key information extraction model combines the content of each word in the text recognition result, the position of each word in the text area, and the image block of the text area in the second image to perform semantic analysis to obtain label information of the equipment and facility; Build a security database, store the tag information of equipment and facilities in the security database, judge the validity period of each equipment and facility in real time based on the tag information of the equipment and facilities, and notify the corresponding administrator when the validity period is less than the set number of days.

[0009] In a second aspect, an embodiment of the present application provides an equipment facility management device based on a text detection and recognition model, comprising: A text detection module is used to obtain a first image, wherein the first image is an image containing equipment labels of equipment facilities, input the first image into a pre-trained text detection model to obtain a second image, perform multi-scale convolution fusion on the first image in the text detection model to obtain a multi-scale fusion result, encode the multi-scale fusion result to obtain a text area mask, perform edge detection on the first image to obtain a text edge, perform cross-attention calculation on the text edge and the text area mask after feature multiplication with the first result to obtain a cross result, perform multiple edge attention calculations on the cross result in sequence to obtain multiple edge attention results, perform feature stacking on the cross result and each edge attention result and output to obtain a second image, wherein the first image is referred to as the first result after edge attention calculation, and the second image is the first image marked with a text area; a label information acquisition module, used to perform text recognition on the second image through a pre-trained text recognition model to obtain a text recognition result, and then input the text recognition result and the second image into a pre-trained key information extraction model, and the key information extraction model combines the content of each word in the text recognition result, the position of each word in the text area, and the image block of the text area in the second image to perform semantic analysis to obtain label information of the equipment and facility; The automatic management module is used to build a security database, store the tag information of equipment and facilities in the security database, judge the validity period of each equipment and facility in real time based on the tag information of the equipment and facilities, and notify the corresponding administrator when the validity period is less than the set number of days.

[0010] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute an equipment facility management method based on a text detection and recognition model.

[0011] In a fourth aspect, an embodiment of the present application provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes an equipment facility management method based on a text detection and recognition model.

[0012] The main contributions and innovations of the present invention are as follows: The embodiment of the present application uses text detection and text recognition technology to automatically obtain label information of equipment and facilities, and manages the validity period of equipment in real time, reducing the need for manual intervention and improving management efficiency; the present solution uses an edge attention mechanism in the text detection model to combine text edge information and text area masks, thereby enhancing the model's ability to locate text areas and further improving the accuracy of text detection; the present solution constructs a key information extraction model to perform semantic analysis on the identified text and position, thereby accurately matching and storing the categories and contents in the label, thereby improving the accuracy of validity period detection.

[0013] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flow chart of a method for equipment and facility management based on a text detection and recognition model according to an embodiment of the present application; Figure 2 is a structural diagram of a text detection model according to an embodiment of the present application; Figure 3 is a structural diagram of a multi-scale convolution and attention module according to an embodiment of the present application; Figure 4 is a structural diagram of an adaptive scale fusion module according to an embodiment of the present application; Figure 5 is a structural diagram of an encoder feature fusion module according to an embodiment of the present application; Figure 6 is a structural diagram of an edge attention module according to an embodiment of the present application; Figure 7 is a structural diagram of a text recognition model according to an embodiment of the present application; Figure 8 is a structural diagram of a key information extraction model implemented according to the present application; Fig. 9 is a logic flow chart of automatic management of the validity period of equipment and facilities according to an embodiment of the present application; Fig.10 It is a structural block diagram of an automatic management device for the validity period of equipment and facilities based on text detection and recognition according to an embodiment of the present application; Fig.11 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0016] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0017] Embodiment 1 The embodiment of the present application provides an equipment facility management method based on a text detection and recognition model, which realizes automatic detection, recognition and validity period management of equipment facility label information through multi-scale convolution fusion, edge attention mechanism and multi-modal Transformer and other technologies, and improves the automation, precision and intelligence level of equipment facility management. Specifically, reference Figure 1 , the method comprising: Acquire a first image, where the first image is an image containing equipment labels of equipment facilities, input the first image into a pre-trained text detection model to obtain a second image, perform multi-scale convolution fusion on the first image in the text detection model to obtain a multi-scale fusion result, encode the multi-scale fusion result to obtain a text area mask, perform edge detection on the first image to obtain a text edge, perform cross-attention calculation on the text edge and the text area mask after feature multiplication with the first result to obtain a cross result, perform multiple edge attention calculations on the cross result in sequence to obtain multiple edge attention results, perform feature stacking on the cross result and each edge attention result and output to obtain a second image, wherein the first image is referred to as the first result after edge attention calculation, and the second image is the first image marked with a text area; Performing text recognition on the second image using a pre-trained text recognition model to obtain a text recognition result, and then inputting the text recognition result and the second image into a pre-trained key information extraction model, wherein the key information extraction model combines the content of each word in the text recognition result, the position of each word in the text area, and the image block of the text area in the second image to perform semantic analysis to obtain label information of the equipment and facility; Build a security database, store the tag information of equipment and facilities in the security database, judge the validity period of each equipment and facility in real time based on the tag information of the equipment and facilities, and notify the corresponding administrator when the validity period is less than the set number of days.

[0018] In some specific embodiments, during the inspection process, the inspection personnel use a mobile phone or other device to photograph the fire-fighting equipment to obtain a photographed image, and then pre-process the photographed image to obtain a first image. The text is detected and recognized in the first image and imported into the security database, so that the validity period of each equipment and facility can be monitored in real time with only one inspection, which greatly reduces the consumption of human resources.

[0019] Specifically, in the preprocessing process, the captured image is first grayscaled to reduce the amount of subsequent data processing, and then the grayscaled image is binarized to further simplify the image information and make the characters clearer. Then, denoising, image size and text direction are adjusted to complete the preprocessing of the captured image.

[0020] In some embodiments, the structure of the text detection model is as follows Figure 2 As shown, in the text detection model, the first image is downsampled by multiple serially connected multi-scale convolution and attention modules, and the downsampling result of each multi-scale convolution and attention module is input into the adaptive scale fusion module to obtain a multi-scale fusion result.

[0021] Specifically, the first image is downsampled through multiple serially connected multi-scale convolution and attention modules to obtain feature maps of different scales. By fusing multiple feature maps of different scales, the text detection model can better restore image information at different levels, especially process blurred parts.

[0022] Furthermore, the structure of the multi-scale convolution and attention module is as follows: Figure 3As shown, the multi-scale convolution and attention module includes a multi-scale convolution unit and an attention unit. In the multi-scale convolution unit, the input of the multi-scale convolution and attention module is first processed sequentially using point-by-point convolution and standard convolution to obtain a first convolution result, and then the first convolution result is respectively input into a plurality of parallel depth-separable hole convolutions of different convolution kernel sizes, and the result of each depth-separable hole convolution is feature stacked and output by point-by-point convolution, and the output result of the point-by-point convolution is residually connected with the first convolution result to obtain a multi-scale convolution result, and then the multi-scale convolution result is input into the attention unit; in the attention unit, the multi-scale convolution result is respectively input into the parallel pixel attention module, channel attention module and simple pixel attention module, and then the output of the pixel attention module, channel attention module and simple pixel attention module is feature stacked and output by point-by-point convolution, and the output result of the point-by-point convolution is residually connected with the multi-scale convolution result as the output result of the multi-scale convolution and attention module.

[0023] Specifically, in the multi-scale convolution unit, the inputs of the multi-scale convolution and attention modules are batch normalized before being processed using point-by-point convolution and standard convolution.

[0024] Specifically, in the attention unit, before the multi-scale convolution results are respectively input into the parallel pixel attention module, the channel attention module and the simple pixel attention module, the multi-scale convolution results are first batch normalized.

[0025] Specifically, in the multi-scale convolution and attention modules, residual connections are used to enable the network to be trained more efficiently and avoid the gradient vanishing problem in deep networks.

[0026] In this scheme, a parallel 7*7 depth-wise separable atrous convolution, a 13*13 depth-wise separable atrous convolution, and a 19*19 depth-wise separable convolution are used in the multi-scale convolution module to process the first convolution result.

[0027] Furthermore, the structure of the adaptive scale fusion module is as follows: Figure 4As shown, in the adaptive scale fusion module, the downsampling results of each multi-scale convolution and attention module are first stacked to obtain a first fusion feature, the first fusion feature is convolved to obtain a second fusion feature, and then the second fusion feature is spatially averaged pooled to obtain a third fusion feature, the third fusion feature is added to the second fusion feature in a channel dimension-upgrading manner to obtain a first fusion output, the first fusion output is then feature-multiplied with the first fusion feature in a channel dimension-upgrading manner to obtain a second fusion output, and the second fusion output is passed through a feature fusion layer to obtain a multi-scale fusion result.

[0028] Specifically, the adaptive scale fusion module learns the weights of feature maps of different scales through stage-level attention to determine the importance of each scale feature in the fusion process, and the attention weights of feature maps in the spatial dimension can be learned through spatial average pooling, so that the network can selectively enhance or suppress features at different spatial positions.

[0029] Specifically, the adaptive scale fusion module allows the network to dynamically select the importance of features of different scales and different positions, so as to more reasonably aggregate the features to enhance scale robustness. In the adaptive scale fusion module, multi-scale features are adaptively fused to enable the network to better process text instances of different sizes. For example, shallow features or large-scale feature maps can capture more detailed information and small text instances, while deep features or small-scale feature maps can capture large-scale text instances and global information. In addition, the adaptive scale fusion module can also enable the network to adapt to different types of text images such as horizontal, multi-directional rotation, and curved shapes.

[0030] In some embodiments, the multi-scale fusion result is input into an encoder feature fusion module for encoding to obtain a text region mask. The structure of the encoder feature fusion module is as follows: Figure 5 As shown, the encoder feature fusion module uses a multi-head self-attention mechanism to process the multi-scale fusion result and then performs linear projection to obtain a projection result. The projection result is first subjected to convolution processing and then feature flattening to obtain a text area mask.

[0031] Specifically, in the encoder feature fusion module, a feature embedding block is first used to convert the multi-scale fusion result into a feature vector representation and then input it into the multi-head self-attention mechanism. The output result of the multi-head self-attention mechanism and the feature vector representation of the multi-scale fusion result are then element-wise added and layer-normalized to obtain a normalized result. The normalized result is input into a linear projection layer to obtain a projection result. The projection result is first convolutionally processed and then feature flattened to obtain a text area mask.

[0032] Specifically, the feature embedding block is used to map the original pixel values ​​in the multi-scale fusion results to a higher-level representation so that the subsequent processing layers can better capture the patterns in the data.

[0033] Specifically, the encoder feature fusion module captures the dependencies between different positions in the input sequence through a multi-head self-attention mechanism, so that information can be learned from different representation subspaces, wherein the multi-head self-attention mechanism includes multiple parallel attention heads, each of which independently calculates the attention weights and then combines these weights to generate the final output.

[0034] Specifically, residual connections are used to perform element-wise addition to preserve the original input information, and layer normalization is used to stabilize the training process to prevent gradient disappearance or explosion.

[0035] Specifically, the normalized result is projected into a new feature space by linear projection to reduce the dimension and readjust the feature representation. The linear projection layer is a linear transformation layer including matrix multiplication and bias terms.

[0036] Specifically, the convolution layer is used to further extract local features in the projection results to enhance the network's ability to understand the local structure of the image. The input of each layer is then standardized through batch normalization and activation functions. Finally, all elements of the feature map are arranged in order into a long vector through feature flattening.

[0037] In some specific embodiments, since the edges of non-text areas may have a negative impact on text segmentation, the edge detection algorithm Canny is used to detect image edges to obtain all text edges in the entire first image.

[0038] In some embodiments, edge attention calculation is performed on the first image and the intersection result by an edge attention module, and the structure of the edge attention module is as follows: Figure 6 As shown, the input of the edge attention module is used as the edge input feature. In the edge attention module, the edge input feature is first feature projected and then sequentially calculated through channel cross-attention and spatial cross-attention to obtain a cross-attention result. The cross-attention result is then output through a feedforward neural network to obtain an edge attention result. The edge attention result obtained by performing edge attention calculation on the first image is the first result.

[0039] Specifically, the edge attention module helps to narrow the semantic gap between the encoder and the decoder by performing cross-attention calculations between the channel cross-attention mechanism and the spatial cross-attention mechanism to capture long-distance dependencies in the channel and spatial dimensions.

[0040] Specifically, the edge input features are first subjected to 2D average pooling and then feature projection, thereby minimizing the computational overhead, so that the network has a low computational cost while maintaining high performance.

[0041] In some specific embodiments, each edge attention result is processed by a multi-layer perceptron, and the processed edge attention result and the cross result are feature stacked, and then the multi-layer perceptron is used to output the feature stacked result to obtain a second image.

[0042] In some specific embodiments, in order to prevent the text mask output by the text detection model from being biased and causing inaccurate marking of the text area in the second image, the text mask output by the text detection model is corrected. Specifically, a set of reference points is predicted by the positioning network, and then the TPS transformation parameters are calculated inside the grid generator based on the reference points. A sampling grid is generated on the second image based on the TPS transformation parameters, and the sampler receives the sampling grid and the first image simultaneously, thereby generating a corrected second image by sampling the sampling grid.

[0043] In some specific embodiments, the training method of the text detection model is: collecting different types of equipment and facility inspection list images as training samples through multiple sources, including but not limited to photos, scans, and electronic screenshots of paper lists, thereby ensuring that these images are taken under different lighting conditions to increase data diversity. During the collection process, it is necessary to ensure that the collected equipment inspection list images have sufficient resolution so that all information that needs to be labeled can be clearly seen, while reducing blur, tilt, and partial occlusion.

[0044] Then use professional annotation tools to annotate each training sample, divide all the labeled training samples into training sets and test sets to train the text detection model to obtain a trained text detection model.

[0045] Furthermore, in order to ensure the adequacy of training samples, data enhancement can be used to enhance the training samples.

[0046] In some specific embodiments, the structure of the text recognition model is as follows: Figure 7 As shown, the text recognition model includes an encoder and a decoder. The encoder extracts the feature representation of the text information and position in the second image through multiple shifted window self-attention mechanisms to obtain shifted text features. The decoder first uses a masked multi-head self-attention mechanism to process the first sequence to obtain first sequence features, and then uses the multi-head self-attention mechanism to identify and predict the first sequence features and the shifted text features to obtain text recognition results, wherein the first sequence is the target sequence generated by the text recognition model.

[0047] Specifically, since the text recognition model is a Transformer architecture, and in the Transformer architecture, the task of the decoder is to gradually generate a target sequence based on the feature vector provided by the encoder. In order to help the decoder better complete this task, in addition to receiving the output from the encoder, it also uses part of the target sequence that has been generated so far as input. The purpose is to allow the decoder to rely only on previously generated words rather than current or subsequent words when predicting the next target at each step when predicting the text sequence. In terms of specific operations, this is achieved by shifting the target sequence as a whole one position to the right, that is, given a target sequence, each of its elements is shifted one position to the right, and a special start marker is added to the beginning of the sequence (such as <bos>). The direct result of this is that at any time point t, the decoder receives all the outputs from time point t-1 and before, ensuring that the generation of the tth word will not be affected by the actual tth word and any information after it, thereby avoiding information leakage. This mechanism is very important for maintaining the autoregressive property, allowing the model to generate each element in the sequence in the correct order. At the same time, this is also part of the teacher forcing technique commonly used in the training process, that is, when generating each step of the sequence, the real target output is used as the input of the next step to help speed up the training process and stabilize learning. However, when actually reasoning or generating text, since the real subsequent output is not available, the output generated by the previous moment itself is used as the input of the next moment.

[0048] Specifically, the shifting window self-attention mechanism divides the image through layered non-overlapping windows, thereby effectively capturing local and global spatial relationships. Each stage in the shifting window self-attention mechanism downsamples the input feature map and extracts features at different scales. This design enables the model to understand the image content at different levels, including details such as characters, symbols, and their relative positions. In addition, the shifting window self-attention mechanism also introduces a shifting window mechanism to enhance information exchange across windows, further improving the expressiveness of the model.

[0049] Specifically, the decoder is a language model of the Transformer architecture. The decoder can automatically generate continuous and grammatically correct sentences based on given prompts. In this scheme, the decoder predicts the various tokens that constitute the text expression in sequence based on the shifted text features given by the encoder until the text characters corresponding to the original image are completely reconstructed. In addition, the decoder in this scheme also uses a multi-head self-attention mechanism to optimize the output results.

[0050] Specifically, the text recognition model is trained using conventional methods. The specific training method can refer to the text detection model, which will not be repeated here.

[0051] In some specific embodiments, the text recognition model in this scheme uses low-rank adaptation technology to fine-tune the pre-trained basic model. During the fine-tuning process, additional learning is performed on the handwritten character dataset, so as to better adapt to handwritten type data while ensuring the original performance of the text recognition model.

[0052] In some specific embodiments, due to the fact that some labels of equipment and facilities may have some inconsistencies when being printed, produced, or recognized by the text recognition model, for example, the expiration date of the validity period may be printed in the device name, etc., this situation is very common. These problems can certainly be found at a glance during manual inspections, but the computer model will not find this problem. If the data with inconsistencies is directly stored in the database, the system will often make misjudgments when judging the validity period. Therefore, this solution constructs a key information extraction model to perform semantic analysis on the recognized text and position to match the category of the equipment label (also the validity period, name, etc.) and related content (specific validity date, specific name, etc.).

[0053] In some embodiments, the structure of the key information extraction model is as follows: Figure 8 As shown, the key information extraction model converts each text content and the position of each text in the text area into a word embedding vector, divides the text area in the second image into image blocks and then converts them into image block embedding vectors, inputs the word embedding vectors into a masked language modeling head, and in the masked language modeling head, character semantics are understood by randomly covering part of the characters to obtain character semantic information, inputs the image block embedding vectors into a masked image modeling head, and in the masked modeling head, image semantics are understood by randomly covering part of the image blocks to obtain image semantic information, and the word embedding vectors and the image block embedding vectors are jointly input into a word block alignment head, and in the word block alignment head, the correspondence between the word embedding vectors and the corresponding image block embedding vectors is learned to obtain alignment information, and the character semantic information, image semantic information and alignment information are integrated to obtain label information of the equipment and facilities.

[0054] Specifically, through the MLM task, the masked language modeling head model can better understand the text content in the document and capture the semantic information of the text, which is crucial for the subsequent key information extraction task because the model needs to accurately understand the meaning of the text in order to accurately identify the key information.

[0055] Specifically, the mask image modeling head passes the MIM task, and the model can better understand the visual information in the document image, such as the document layout, font, color, etc. This is very important for the key information extraction task, because the visual layout of the document (such as the layout of tables and titles) can often provide important clues to help the model identify key information more accurately.

[0056] Specifically, the purpose of the block alignment header is to establish a fine-grained correspondence between text and images. In document understanding tasks, the layout and position information of the text is very important. Through the WPA task, the model can associate the text content with its spatial position in the document. This is very important for key information extraction tasks because the model needs to judge the importance of the text based on its position information (for example, the content in the title, body, table, etc.).

[0057] Furthermore, the label information of the equipment and facilities is structured data.

[0058] Specifically, the key information extraction model in this scheme can consider both the content of the text and their relative positions on the page to generate a richer semantic representation. It directly utilizes the image blocks of the document image, greatly saving parameters and avoiding complex document preprocessing steps, so that documents can be processed more efficiently and more easily adapted to different application scenarios.

[0059] In some specific embodiments, the validity period of all stored tag information can be judged in the security database to ensure that every equipment and facility in the entire security area will not expire. When judging the validity period, if the equipment validity period is directly written in the tag information, the equipment validity period can be directly compared with the current date. If the tag information contains the shelf life of the equipment, then the production date of the equipment is added with the shelf life and then compared with the current date.

[0060] It is worth mentioning that managers can also directly query the validity period or other information of each equipment and facility in the security database.

[0061] In some specific embodiments, the process of automatic management of the validity period of equipment and facilities in this solution is as follows Fig. 9 As shown, first set up an inspection area, mark the location of each equipment and facility in the inspection area, arrange inspectors to inspect each equipment and facility in the inspection area, use mobile phones or other devices to capture the first image of each equipment and facility during the inspection and upload it to the system. If it is found during the first inspection that the equipment and facilities such as fire extinguishers have expired bottles or bottles, they will be replaced directly, and then the first image of the replaced equipment and facilities will be captured and uploaded. If everything is normal, the system will directly identify the information in the first image and save it to the electronic ledger, and then upload it to the security database for storage.

[0062] Embodiment 2 Based on the same idea, refer to Fig.10 , the present application also proposes an equipment facility management device based on a text detection and recognition model, comprising: A text detection module is used to obtain a first image, wherein the first image is an image containing equipment labels of equipment facilities, input the first image into a pre-trained text detection model to obtain a second image, perform multi-scale convolution fusion on the first image in the text detection model to obtain a multi-scale fusion result, encode the multi-scale fusion result to obtain a text area mask, perform edge detection on the first image to obtain a text edge, perform cross-attention calculation on the text edge and the text area mask after feature multiplication with the first result to obtain a cross result, perform multiple edge attention calculations on the cross result in sequence to obtain multiple edge attention results, perform feature stacking on the cross result and each edge attention result and output to obtain a second image, wherein the first image is referred to as the first result after edge attention calculation, and the second image is the first image marked with a text area; a label information acquisition module, used to perform text recognition on the second image through a pre-trained text recognition model to obtain a text recognition result, and then input the text recognition result and the second image into a pre-trained key information extraction model, and the key information extraction model combines the content of each word in the text recognition result, the position of each word in the text area, and the image block of the text area in the second image to perform semantic analysis to obtain label information of the equipment and facility; The automatic management module is used to build a security database, store the tag information of equipment and facilities in the security database, judge the validity period of each equipment and facility in real time based on the tag information of the equipment and facilities, and notify the corresponding administrator when the validity period is less than the set number of days.

[0063] Embodiment 3 This embodiment also provides an electronic device, referring to Fig.11 , comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0064] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0065] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. For example, but not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. In appropriate circumstances, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0066] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .

[0067] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any one of the equipment facility management methods based on the text detection and recognition model in the above embodiments.

[0068] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .

[0069] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0070] The input / output device 408 is used to input or output information. In this embodiment, the input information may be an image containing a device label of a device facility, and the output information may be label information of the device facility.

[0071] Optionally, in this embodiment, the processor 402 may be configured to perform the following steps through a computer program: Acquire a first image, where the first image is an image containing equipment labels of equipment facilities, input the first image into a pre-trained text detection model to obtain a second image, perform multi-scale convolution fusion on the first image in the text detection model to obtain a multi-scale fusion result, encode the multi-scale fusion result to obtain a text area mask, perform edge detection on the first image to obtain a text edge, perform cross-attention calculation on the text edge and the text area mask after feature multiplication with the first result to obtain a cross result, perform multiple edge attention calculations on the cross result in sequence to obtain multiple edge attention results, perform feature stacking on the cross result and each edge attention result and output to obtain a second image, wherein the first image is referred to as the first result after edge attention calculation, and the second image is the first image marked with a text area; Performing text recognition on the second image using a pre-trained text recognition model to obtain a text recognition result, and then inputting the text recognition result and the second image into a pre-trained key information extraction model, wherein the key information extraction model combines the content of each word in the text recognition result, the position of each word in the text area, and the image block of the text area in the second image to perform semantic analysis to obtain label information of the equipment and facility; Build a security database, store the tag information of equipment and facilities in the security database, judge the validity period of each equipment and facility in real time based on the tag information of the equipment and facilities, and notify the corresponding administrator when the validity period is less than the set number of days.

[0072] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0073] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0074] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Fig.11 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0075] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.< / bos>

Claims

1. A method for equipment and facility management based on a text detection and recognition model, characterized in that: The following steps are involved: Acquire a first image, where the first image is an image containing equipment labels of equipment facilities, input the first image into a pre-trained text detection model to obtain a second image, perform multi-scale convolution fusion on the first image in the text detection model to obtain a multi-scale fusion result, encode the multi-scale fusion result to obtain a text area mask, perform edge detection on the first image to obtain a text edge, perform cross-attention calculation on the text edge and the text area mask after feature multiplication with the first result to obtain a cross result, perform multiple edge attention calculations on the cross result in sequence to obtain multiple edge attention results, perform feature stacking on the cross result and each edge attention result and output to obtain a second image, wherein the first image is referred to as the first result after edge attention calculation, and the second image is the first image marked with a text area; Performing text recognition on the second image using a pre-trained text recognition model to obtain a text recognition result, and then inputting the text recognition result and the second image into a pre-trained key information extraction model, wherein the key information extraction model combines the content of each word in the text recognition result, the position of each word in the text area, and the image block of the text area in the second image to perform semantic analysis to obtain label information of the equipment and facility; Build a security database, store the tag information of equipment and facilities in the security database, judge the validity period of each equipment and facility in real time based on the tag information of the equipment and facilities, and notify the corresponding administrator when the validity period is less than the set number of days.

2. The equipment facility management method based on text detection and recognition model according to claim 1 is characterized in that: In the text detection model, the first image is downsampled through multiple serially connected multi-scale convolution and attention modules, and the downsampling result of each multi-scale convolution and attention module is input into an adaptive scale fusion module to obtain a multi-scale fusion result.

3. The equipment facility management method based on text detection and recognition model according to claim 1 is characterized in that: The multi-scale fusion result is input into the encoder feature fusion module for encoding to obtain a text area mask. The encoder feature fusion module uses a multi-head self-attention mechanism to process the multi-scale fusion result and then performs linear projection to obtain a projection result. The projection result is first subjected to convolution processing and then feature flattening to obtain a text area mask.

4. The equipment facility management method based on text detection and recognition model according to claim 1 is characterized in that: An edge attention module is used to perform edge attention calculation on the first image and the cross result. The input of the edge attention module is used as the edge input feature. In the edge attention module, the edge input feature is first feature projected and then sequentially subjected to channel cross attention and spatial cross attention calculations to obtain a cross-attention result. The cross-attention result is then output through a feedforward neural network to obtain an edge attention result. The edge attention result obtained by performing edge attention calculation on the first image is the first result.

5. The equipment facility management method based on text detection and recognition model according to claim 1 is characterized in that: The text recognition model includes an encoder and a decoder. The encoder extracts the feature representation of the text information and position in the second image through multiple shifted window self-attention mechanisms to obtain shifted text features. The decoder first uses a masked multi-head self-attention mechanism to process the first sequence to obtain first sequence features, and then uses the multi-head self-attention mechanism to identify and predict the first sequence features and the shifted text features to obtain text recognition results, wherein the first sequence is a target sequence generated by the text recognition model.

6. The equipment facility management method based on text detection and recognition model according to claim 1 is characterized in that: The key information extraction model converts each text content and the position of each text in the text area into a word embedding vector, divides the text area in the second image into image blocks and converts them into image block embedding vectors, inputs the word embedding vectors into a masked language modeling head, and in the masked language modeling head, character semantics are understood by randomly covering part of the characters to obtain character semantic information, inputs the image block embedding vectors into a masked image modeling head, and in the masked modeling head, image semantics are understood by randomly covering part of the image blocks to obtain image semantic information, and the word embedding vectors and the image block embedding vectors are jointly input into a word block alignment head, and in the word block alignment head, the correspondence between the word embedding vectors and the corresponding image block embedding vectors is learned to obtain alignment information, and the character semantic information, image semantic information and alignment information are integrated to obtain label information of the equipment and facilities.

7. The equipment facility management method based on text detection and recognition model according to claim 1 is characterized in that: The label information of equipment and facilities is structured data.

8. An equipment facility management device based on a text detection and recognition model, characterized in that: include: A text detection module is used to obtain a first image, wherein the first image is an image containing equipment labels of equipment facilities, input the first image into a pre-trained text detection model to obtain a second image, perform multi-scale convolution fusion on the first image in the text detection model to obtain a multi-scale fusion result, encode the multi-scale fusion result to obtain a text area mask, perform edge detection on the first image to obtain a text edge, perform cross-attention calculation on the text edge and the text area mask after feature multiplication with the first result to obtain a cross result, perform multiple edge attention calculations on the cross result in sequence to obtain multiple edge attention results, perform feature stacking on the cross result and each edge attention result and output to obtain a second image, wherein the first image is referred to as the first result after edge attention calculation, and the second image is the first image marked with a text area; a label information acquisition module, used to perform text recognition on the second image through a pre-trained text recognition model to obtain a text recognition result, and then input the text recognition result and the second image into a pre-trained key information extraction model, and the key information extraction model combines the content of each word in the text recognition result, the position of each word in the text area, and the image block of the text area in the second image to perform semantic analysis to obtain label information of the equipment and facility; The automatic management module is used to build a security database, store the tag information of equipment and facilities in the security database, judge the validity period of each equipment and facility in real time based on the tag information of the equipment and facilities, and notify the corresponding administrator when the validity period is less than the set number of days.

9. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the equipment facility management method based on a text detection and recognition model as described in any one of claims 1-7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, and the process includes an equipment facility management method based on a text detection and recognition model according to any one of claims 1-7.

Images