Digitalized reconstruction method, device and equipment for stone tablet image and medium
By extracting corner maps and character content description information of the stele image, deep feature extraction and prediction are performed, combined with the optimization of the fusion model, the problem of low accuracy of the stele image reconstruction is solved, and the reconstruction of the stele image with high accuracy is achieved.
Patent Information
- Application Number
- CN202411057102.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-02
AI Technical Summary
The existing methods of stone tablet image reconstruction have the problem of low reconstruction accuracy, especially when dealing with fuzzy and degraded stone tablet images, it is difficult to maintain the integrity of the internal structure information of characters and the global features of the image.
By obtaining corner maps and character content description information of multiple stone tablet images, shallow features are extracted and deep feature extraction is performed, predicting predicted characters and obtaining image prior knowledge, and fusion feature information is used to construct reconstruction loss functions and boundary losses by fusion of fusion models. 函数,对模型进行优化,最终生成高准确性的重建图像。
The reconstruction accuracy and information richness of the stele image are improved, ensuring the accuracy and feature integrity of the final reconstruction image.
Smart Images

Figure CN118941668A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image reconstruction technology, and in particular to a method, device, equipment and medium for digitally reconstructing a stone tablet image. Background Art
[0002] Stone monuments have been eroded by various natural factors such as weathering and earthquakes throughout history, which has led to the gradual loss of their structure and surface details, and caused varying degrees of degradation such as blurring and fractures. With the development of scanning technology and artificial intelligence technology, researchers use digital reconstruction technology to digitally model and reconstruct images of cultural relics, which is not only beneficial to the protection of stone monuments themselves, but also provides valuable historical data for academic research. Scholars can conduct detailed text and image analysis through digital cultural relics, thereby conducting in-depth research on the cultural, economic and technological levels of ancient society.
[0003] In the process of digital reconstruction of stele images, due to the blur and degradation of the stele images themselves, there is a certain connection and overlap between the text characters in the image and the background pattern, and the shape and outline of the characters are complex and not clear enough. Many studies have not been able to effectively maintain the structural information inside the characters and the integrity of the global features of the image when dealing with the relationship between the characters and the image background. It can be seen that the existing stele image reconstruction methods have the problem of low reconstruction accuracy of stele images. Summary of the invention
[0004] The present application provides a method, device, equipment and medium for digitally reconstructing a stele image, which can solve the problem of low reconstruction accuracy of the stele image.
[0005] In a first aspect, an embodiment of the present application provides a method for digitally reconstructing a stele image, the method comprising:
[0006] Acquire multiple stele images, and extract the corner point map and character content description information of each stele image; the corner point map is an image that marks the corner points of all characters in the stele image;
[0007] For each stele image, shallow features of the stele image are obtained according to the stele image and the corner point map of the stele image, and deep features are extracted from the shallow image features to obtain deep features of the stele image;
[0008] For each deep feature, the deep feature is predicted to obtain the predicted character of the stone tablet image corresponding to the deep feature, and the image prior knowledge of the predicted character is obtained; the image prior knowledge is used to describe the image information generated according to the predicted character;
[0009] The fusion model is used to fuse the shallow features, image prior knowledge, and character content description information corresponding to each stele image to obtain the final features of each stele image.
[0010] Reconstruct each final feature using the reconstruction model to obtain a reconstructed image of the stone tablet image corresponding to each final feature;
[0011] A reconstruction loss function and a pixel loss function are constructed based on all reconstructed images, and a boundary loss function is constructed based on all final features. The reconstruction loss function, the pixel loss function and the boundary loss function are used to optimize the fusion model and the reconstruction model to obtain an optimized fusion model and an optimized reconstruction model. The reconstruction loss function and the pixel loss function are used to describe the accuracy of all reconstructed images, and the boundary loss function is used to describe the accuracy of all final features.
[0012] The stone tablet image to be reconstructed is reconstructed using the optimized fusion model and the optimized reconstruction model to obtain a final reconstructed image of the stone tablet image to be reconstructed.
[0013] Optionally, shallow features of the stone tablet image are obtained according to the stone tablet image and the corner point map of the stone tablet image, including:
[0014] By formula:
[0015] R m,2 ,C2=UNet(R m,1 ,C1)
[0016] R m,1 ,C1=ResNet(R m,0 ,C0)
[0017] R m,0 ,C0=Init(I stone,m ,Cor stone,m )
[0018] Get the shallow feature R of the mth stele image m,2 ;
[0019] Among them, C2, C1, and C0 all represent the number of feature map channels, UNet() represents the UNet network operation, and R m,1 Represents secondary features, ResNet() represents ResNet network operation, R m,0 Indicates the initial features, Init() indicates the convolution layer and pooling layer operations, I stone,m represents the mth stele image, Cor stone,m represents the corner point map of the mth stele image, m = 1, 2, ..., M, and M represents the total number of stele images.
[0020] Optionally, a fusion model is used to fuse the shallow features, image prior knowledge, and character content description information corresponding to each stele image to obtain the final features of each stele image, including:
[0021] For each stele image, perform the following steps:
[0022] The shallow features corresponding to the stele image and the corner map are fused to obtain the initial fusion features, and the initial fusion features are fused with the image prior knowledge corresponding to the stele image to obtain the secondary fusion features;
[0023] Feature extraction is performed on the character content description information corresponding to the stele image to obtain text features, and the text features are fused with the secondary fusion features to obtain the final features of the stele image.
[0024] Optionally, each final feature is reconstructed using a reconstruction model to obtain a reconstructed image of the stone tablet image corresponding to each final feature, including:
[0025] By formula:
[0026] R m,SR =Pix(R m,2 +RL m,cross )
[0027] Get the reconstructed image R of the mth stele image m,SR ;
[0028] Among them, Pix() represents upsampling, R m,2 represents the shallow features of the m-th stele image, RL m,cross represents the final feature of the mth stele image, m = 1, 2, ..., M, and M represents the total number of stele images.
[0029] Optionally, a reconstruction loss function and a pixel loss function are constructed based on all reconstructed images, including:
[0030] Perform text recognition on each reconstructed image to obtain the text recognition result of each reconstructed image, and construct a reconstruction loss function based on all the text recognition results;
[0031] A pixel loss function is constructed based on all reconstructed images.
[0032] Optionally, the reconstruction loss function is:
[0033] l rec =1 / m∑(pre sr,m -pre hr,m ) 2
[0034] Among them, l recRepresents the value of the reconstruction loss function, pre sr,m represents the text recognition result of the mth reconstructed image, pre hr,m Represents the true text label of the m-th stele image;
[0035] The pixel loss function is:
[0036]
[0037] Among them, l2 represents the value of the pixel loss function, represents the pixel value of the i-th pixel of the m-th reconstructed image, Represents the pixel value of the i-th pixel point of the m-th stone tablet image.
[0038] Optionally, the boundary loss function is:
[0039]
[0040] Among them, l box represents the value of the boundary loss function, t m represents the bounding box parameters obtained according to the final features corresponding to the m-th stele image, Represents the ground-truth bounding box parameters of the m-th stele image.
[0041] In a second aspect, an embodiment of the present application provides a device for digitally reconstructing a stele image, comprising:
[0042] An acquisition module acquires multiple stele images and extracts a corner point map and character content description information of each stele image; the corner point map is an image that marks the corner points of all characters in the stele image;
[0043] The feature extraction module obtains shallow features of each stele image according to the stele image and the corner map of the stele image, and extracts deep features from the shallow image features to obtain deep features of the stele image;
[0044] The prediction module predicts each deep feature, obtains the predicted character of the stone tablet image corresponding to the deep feature, and obtains the image prior knowledge of the predicted character; the image prior knowledge is used to describe the image information generated according to the predicted character;
[0045] The fusion module uses the fusion model to fuse the shallow features, image prior knowledge, and character content description information corresponding to each stele image to obtain the final features of each stele image;
[0046] A first reconstruction module reconstructs each final feature using a reconstruction model to obtain a reconstructed image of the stone tablet image corresponding to each final feature;
[0047] The optimization module constructs a reconstruction loss function and a pixel loss function based on all reconstructed images, constructs a boundary loss function based on all final features, and optimizes the fusion model and the reconstruction model using the reconstruction loss function, the pixel loss function and the boundary loss function to obtain an optimized fusion model and an optimized reconstruction model; the reconstruction loss function and the pixel loss function are used to describe the accuracy of all reconstructed images, and the boundary loss function is used to describe the accuracy of all final features;
[0048] The second reconstruction module reconstructs the to-be-reconstructed stone tablet image using the optimized fusion model and the optimized reconstruction model to obtain a final reconstructed image of the to-be-reconstructed stone tablet image.
[0049] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for digitally reconstructing the stone tablet image when executing the above-mentioned computer program.
[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned method for digital reconstruction of the stone tablet image.
[0051] The above solution of the present application has the following beneficial effects:
[0052] In some embodiments of the present application, a plurality of stele images are acquired, and a corner point map and character content description information of each stele image are extracted. Then, for each stele image, shallow features of the stele image are obtained according to the stele image and the corner point map of the stele image, and deep features are extracted from the shallow image features to obtain deep features of the stele image. Then, for each deep feature, the deep feature is predicted to obtain the predicted character of the stele image corresponding to the deep feature, and the image prior knowledge of the predicted character is obtained. Then, a fusion model is used to perform the shallow features, image prior knowledge, and character content description information corresponding to each stele image. Fusion is performed to obtain the final features of each stele image, and then each final feature is reconstructed using a reconstruction model to obtain a reconstructed image of the stele image corresponding to each final feature, and then a reconstruction loss function and a pixel loss function are constructed based on all reconstructed images, and a boundary loss function is constructed based on all final features, and the fusion model and the reconstruction model are optimized using the reconstruction loss function, the pixel loss function and the boundary loss function to obtain an optimized fusion model and an optimized reconstruction model, and finally the stele image to be reconstructed is reconstructed using the optimized fusion model and the optimized reconstruction model to obtain the final reconstructed image of the stele image to be reconstructed. Among them, obtaining a corner point map of the stele image can provide corner point information of the stele image, so that the information richness of the deep features obtained based on the corner point information is improved, thereby improving the accuracy of the acquired image prior knowledge, and the accuracy and information richness of the final features obtained by fusion based on accurate image prior knowledge, character content description information, and shallow features are improved, and the accuracy of the reconstructed image obtained by reconstruction based on the final features with high accuracy and richness is improved.
[0053] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0055] Figure 1 A flowchart of a method for digitally reconstructing a stele image provided in one embodiment of the present application;
[0056] Figure 2 A schematic diagram of the structure of a device for digitally reconstructing a stele image provided in one embodiment of the present application;
[0057] Figure 3 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0058] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0059] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0060] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0061] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0062] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0063] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0064] In order to solve the problem of low accuracy in the existing reconstruction of stone tablet images, an embodiment of the present application provides a method for digitally reconstructing stone tablet images. The method obtains multiple stone tablet images and extracts a corner point map and character content description information of each stone tablet image. Then, for each stone tablet image, shallow features of the stone tablet image are obtained based on the stone tablet image and the corner point map of the stone tablet image. Deep features are extracted from the shallow image features to obtain deep features of the stone tablet image. Then, for each deep feature, the deep feature is predicted to obtain the predicted character of the stone tablet image corresponding to the deep feature. The image prior knowledge of the predicted character is obtained. Then, a fusion model is used to predict the predicted character corresponding to each stone tablet image. The shallow features, image prior knowledge, and character content description information are fused to obtain the final features of each stele image, and then the reconstruction model is used to reconstruct each final feature to obtain the reconstructed image of the stele image corresponding to each final feature, and then the reconstruction loss function and pixel loss function are constructed based on all the reconstructed images, and the boundary loss function is constructed based on all the final features, and the fusion model and the reconstruction model are optimized using the reconstruction loss function, pixel loss function, and boundary loss function to obtain the optimized fusion model and the optimized reconstruction model, and finally the optimized fusion model and the optimized reconstruction model are used to reconstruct the stele image to be reconstructed to obtain the final reconstructed image of the stele image to be reconstructed. Among them, obtaining the corner point map of the stele image can provide the corner point information of the stele image, so that the information richness of the deep features obtained based on the corner point information is improved, thereby improving the accuracy of the acquired image prior knowledge, and the accuracy and information richness of the final features obtained by fusion of accurate image prior knowledge, character content description information, and shallow features are improved, and the accuracy of the reconstructed image obtained by reconstruction based on the final features with high accuracy and richness is improved.
[0065] Next, the digital reconstruction method of the stone tablet image provided in this application is exemplified.
[0066] like Figure 1 As shown, the digital reconstruction method of the stone tablet image provided by the present application includes the following steps:
[0067] Step 11, obtaining multiple stele images, and extracting the corner point map and character content description information of each stele image.
[0068] The above corner point map is an image in which the corner points of all characters in the stele image are annotated, that is, a stele image with corner point annotations. The above stele image is an image of a stele, such as a historical memorial stele, an architectural memorial stele, a calligraphy stele, and the like. The character content description information is used to describe the content recorded by all characters in the stele image. For example, for a stele image of an architectural memorial stele, the character content description information may be "the stele records the completion time of the building, the name of the investor, the time and cost."
[0069] In some embodiments of the present application, a camera or other device may be used to obtain a stele image. In order to increase the diversity of the stele image, data enhancement techniques such as rotation, scaling, and cropping may be used to process the directly obtained stele image to generate multiple stele images. An algorithm for detecting corner points such as the Harris corner detection algorithm may be used to obtain a corner point map of each stele image. Character content description information of the stele image may be extracted manually.
[0070] It should be noted that after obtaining the character content description information, the character content description information can be preprocessed, such as using the jieba word segmentation tool to perform word segmentation, dividing the text description into a sequence of words, and then filtering the words in the word sequence according to the stop word list of Harbin Institute of Technology, deleting and filtering out the words in the stop word list, and finally comparing the part of speech marked by the jieba tool during word segmentation with the custom reserved part of speech table, deleting and filtering out the words whose part of speech is not in the reserved part of speech table.
[0071] It is worth mentioning that by obtaining the corner point map and character content description information of the stele image, the corner point information of the stele image and the information of the recorded content can be provided, thereby improving the richness of the information.
[0072] Step 12, for each stele image, obtain shallow features of the stele image according to the stele image and the corner point map of the stele image, and perform deep feature extraction on the shallow image features to obtain deep features of the stele image.
[0073] In some embodiments of the present application, the above-mentioned step of obtaining shallow features of the stone tablet image based on the stone tablet image and the corner point map of the stone tablet image, and extracting deep features from the shallow image features to obtain deep features of the stone tablet image includes:
[0074] The first step is to obtain the shallow features of the stone tablet image based on the stone tablet image and the corner point map of the stone tablet image.
[0075] Specifically, through the formula:
[0076] R m,2 ,C2=UNet(R m,1 ,C1)
[0077] R m,1 ,C1=ResNet(R m,0 ,C0)
[0078] R m,0 ,C0=Init(I stone,m ,Cor stone,m )
[0079] Get the shallow feature R of the mth stele image m,2.
[0080] Among them, C2, C1, and C0 all represent the number of feature map channels, UNet() represents the UNet network operation, and R m,1 Represents secondary features, ResNet() represents ResNet network operation, R m,0 Indicates the initial features, Init() indicates the convolution layer and pooling layer operations, I stone,m represents the mth stele image, Cor stone,m represents the corner point map of the mth stele image, m = 1, 2, ..., M, and M represents the total number of stele images.
[0081] For example, the above formula can be run using computer software such as matlab and python to obtain shallow features.
[0082] The second step is to extract deep features from shallow image features to obtain the deep features of the stone tablet image.
[0083] Exemplarily, Transformer can be used to extract deep features from shallow image features to obtain deep features.
[0084] It is worth mentioning that the shallow features and deep features obtained based on the corner map incorporate the corner information of the stone tablet image, which can improve the accuracy and information richness of the shallow features and deep features.
[0085] Step 13, predicting each deep feature respectively, obtaining the predicted character of the stone tablet image corresponding to the deep feature, and acquiring the image prior knowledge of the predicted character.
[0086] The above-mentioned image prior knowledge is used to describe the image information generated according to the predicted characters.
[0087] In some embodiments of the present application, the deep features can be processed step by step using a linear layer and a softmax layer to obtain a predicted character. The predicted character is a character in the stone tablet image corresponding to the deep features predicted based on the deep features. The above steps of obtaining the image prior knowledge of the predicted character include:
[0088] The contrastive language-image pre-training model (CLIP) can be used to vectorize the content description information of each character in existing literature. The text is decomposed into sub-word units through a word segmenter and mapped to their corresponding word embedding vectors to form a text input sequence. Since the CLIP model has powerful visual and semantic understanding capabilities, it can learn the relationship between text and images. According to the information described by each text input sequence, the CLIP model can be used to generate the image corresponding to each text input vector. All combinations of text input vectors and corresponding images are saved in the same database to obtain a priori knowledge base.
[0089] Then, an image matching the predicted character is searched in the prior knowledge base, such as calculating the cosine similarity between each text input vector in the prior knowledge base and the predicted character, selecting the text input vector corresponding to the smallest cosine similarity value, and using the image corresponding to the text input vector as the image matching the predicted character. A convolutional neural network is used to obtain an image feature vector of the image, and the image feature vector is used as the image prior knowledge of the corresponding predicted character.
[0090] Step 14, using the fusion model to fuse the shallow features, image prior knowledge, and character content description information corresponding to each stele image to obtain the final features of each stele image.
[0091] In some embodiments of the present application, the step of using the fusion model to fuse the shallow features, image prior knowledge, and character content description information corresponding to each stele image to obtain the final features of each stele image includes:
[0092] The fusion model performs the following steps for each stele image:
[0093] In the first step, the shallow features corresponding to the stele image and the corner map are fused to obtain the initial fusion features, and the initial fusion features are fused with the image prior knowledge corresponding to the stele image to obtain the secondary fusion features.
[0094] Exemplarily, the cross-attention mechanism can be used for fusion, that is, the corner map is used as the query vector in the cross-attention mechanism, the shallow features are used as the key vector and the value vector, and the initial fused features are obtained after the cross-attention calculation. Then, the self-attention mechanism is used to calculate the attention score of the image prior knowledge, and the attention score is used as the query vector in the cross-attention mechanism, and the initial fused features are used as the key vector and the value vector, and the secondary fused features are obtained after the cross-attention calculation.
[0095] The second step is to extract the character content description information corresponding to the stele image to obtain text features, and then fuse the text features with the secondary fusion features to obtain the final features of the stele image.
[0096] Exemplarily, the encoder can be used to extract features from character content description information to obtain text features, and the cross-attention mechanism can be used to fuse the text features and secondary fusion features, that is, the secondary fusion features are used as query vectors of the cross-attention mechanism, and the text features are used as key vectors and value vectors to perform cross-attention calculations to obtain the final features.
[0097] The above cross attention is calculated as:
[0098]
[0099]
[0100]
[0101]
[0102] Among them, Q i represents the query vector, K t represents the key vector, V t represents the value vector, I represents the input as the query vector, T represents the input as the key vector and the value vector, and Both represent weights, d k Represents dimension.
[0103] It can be understood that the structure of the fusion model is a self-attention module, a feature extraction module, and three fusion modules connected in sequence. The input of the self-attention module inputs the image prior knowledge, the output of the self-attention module is connected to the input of the second fusion module, the input of the feature extraction module inputs the character content description information, the output of the feature extraction module is connected to the input of the third fusion module, the input of the first fusion module is the corner map and the shallow feature, and the output is the initial fusion feature, the input of the second fusion module is the initial fusion feature and the attention score obtained by the self-attention module, and the output is the secondary fusion feature, the input of the third fusion module is the text feature obtained by the feature extraction module and the secondary fusion feature, and the output is the final feature. The calculation mechanism in each fusion module is related to the actual fusion mechanism adopted. If the three fusion modules are all fused using the cross-attention mechanism, the calculation formula of the cross-attention mechanism can be considered to be the expression of the three fusion modules.
[0104] It is worth mentioning that the accuracy and information richness of the final features obtained by fusion of accurate image prior knowledge, character content description information, and shallow features are improved.
[0105] Step 15, reconstruct each final feature using the reconstruction model to obtain a reconstructed image of the stone tablet image corresponding to each final feature.
[0106] Specifically, the reconstruction model is based on the formula:
[0107] R m,SR =Pix(R m,2 +RL m,cross )
[0108] Get the reconstructed image R of the mth stele image m,SR .
[0109] Among them, Pix() represents upsampling, R m,2 represents the shallow features of the m-th stele image, RL m,cross represents the final feature of the mth stele image, m = 1, 2, ..., M, and M represents the total number of stele images.
[0110] It can be understood that the above formula for obtaining the reconstructed image is an expression of the reconstruction model.
[0111] Exemplarily, the above formula can be run using computer software such as matlab, python, etc. to obtain a reconstructed image.
[0112] It is worth mentioning that the accuracy of the reconstructed image is improved by reconstructing based on the final features with high accuracy and richness.
[0113] Step 16, constructing a reconstruction loss function and a pixel loss function based on all reconstructed images, constructing a boundary loss function based on all final features, and optimizing the fusion model and the reconstruction model using the reconstruction loss function, the pixel loss function and the boundary loss function to obtain an optimized fusion model and an optimized reconstruction model.
[0114] The above reconstruction loss function and pixel loss function are used to describe the accuracy of all reconstructed images, and the boundary loss function is used to describe the accuracy of all final features.
[0115] In some embodiments of the present application, the steps of constructing a reconstruction loss function and a pixel loss function based on all reconstructed images, constructing a boundary loss function based on all final features, and optimizing the fusion model and the reconstruction model using the reconstruction loss function, the pixel loss function and the boundary loss function to obtain the optimized fusion model and the optimized reconstruction model include:
[0116] In the first step, a reconstruction loss function and a pixel loss function are constructed based on all reconstructed images.
[0117] First, text recognition is performed on each reconstructed image to obtain the text recognition result of each reconstructed image, and a reconstruction loss function is constructed based on all text recognition results.
[0118] Specifically, the reconstruction loss function is:
[0119] l rec =1 / m∑(pre sr,m -pre hr,m ) 2
[0120] Among them, l rec Represents the value of the reconstruction loss function, pre sr,m represents the text recognition result of the mth reconstructed image, pre sr,m Represents the true text label of the m-th stele image.
[0121] Then, a pixel loss function is constructed based on all reconstructed images.
[0122] Specifically, the pixel loss function is:
[0123]
[0124] Among them, l2 represents the value of the pixel loss function, represents the pixel value of the i-th pixel of the m-th reconstructed image, Represents the pixel value of the i-th pixel point of the m-th stone tablet image.
[0125] Exemplarily, a convolutional recurrent neural network can be used to perform text recognition on the reconstructed image.
[0126] It should be noted that the above-mentioned real text labels are used to describe the real text on the stone tablet corresponding to the stone tablet image.
[0127] In the second step, a boundary loss function is constructed based on all the final features.
[0128] Specifically, the boundary loss function is:
[0129]
[0130] Among them, l box represents the value of the boundary loss function, t m represents the bounding box parameters obtained according to the final features corresponding to the m-th stele image, Represents the ground-truth bounding box parameters of the m-th stele image.
[0131] In the third step, the fusion model and the reconstruction model are optimized using the reconstruction loss function, the pixel loss function and the boundary loss function to obtain the optimized fusion model and the optimized reconstruction model.
[0132] Specifically, determine whether the value of the reconstruction loss function, the value of the pixel loss function and the value of the boundary loss function are all less than the preset loss function value. If so, use the fusion model as the optimized fusion model, and use the reconstruction model as the optimized reconstruction model. Otherwise, adjust the parameters of the fusion model and the reconstruction model, and return to the step of using the fusion model to fuse the shallow features, image prior knowledge, and character content description information corresponding to each stone tablet image to obtain the final features of each stone tablet image.
[0133] Exemplarily, the preset loss function value is 1.8, the values of the reconstruction loss function, the pixel loss function and the boundary loss function are 2.1, 1.3 and 1.5 respectively, where the value of the reconstruction loss function is greater than the preset loss function value, the parameters of the fusion model and the reconstruction model are adjusted, and the step of using the fusion model to fuse the shallow features, image prior knowledge, and character content description information corresponding to each stele image to obtain the final features of each stele image is returned, and the values of the reconstruction loss function, the pixel loss function and the boundary loss function are re-obtained. At this time, the values are 1.4, 1.2 and 1.6 respectively, which are all less than 1.8. The fusion model at this time is used as the optimized fusion model, and the reconstruction model is used as the optimized reconstruction model.
[0134] Step 17, reconstructing the stone tablet image to be reconstructed using the optimized fusion model and the optimized reconstruction model to obtain a final reconstructed image of the stone tablet image to be reconstructed.
[0135] The above-mentioned stone tablet image to be reconstructed is a stone tablet image that needs to be reconstructed.
[0136] Specifically, the corner point map and character content description information of the stone tablet image to be reconstructed are extracted, and then the stone tablet image to be reconstructed and the corner point map are used to obtain shallow features and deep features, and then the image prior knowledge is obtained based on the deep features. Then, the shallow features, image prior knowledge, and character content description information are fused using the optimized fusion model to obtain the final features of the stone tablet image to be reconstructed. Finally, the final features are reconstructed using the optimized reconstruction model to obtain the final reconstructed image of the stone tablet image to be reconstructed.
[0137] Exemplarily, the above steps can be run using computer software such as matlab and python to obtain the final reconstructed image of the stone tablet image to be reconstructed.
[0138] It is worth mentioning that obtaining the corner point map of the stele image can provide the corner point information of the stele image, thereby improving the information richness of the deep features obtained based on the corner point information, and then improving the accuracy of the acquired image prior knowledge. The accuracy and information richness of the final features obtained by fusion of accurate image prior knowledge, character content description information, and shallow features are improved, and the accuracy of the reconstructed image obtained by reconstruction based on the final features with high accuracy and richness is improved.
[0139] The following is an exemplary description of the digital reconstruction device for the stone tablet image provided in this application.
[0140] like Figure 2 As shown, the embodiment of the present application provides a digital reconstruction device for a stele image, and the digital reconstruction device 200 includes:
[0141] The acquisition module 201 acquires a plurality of stele images and extracts a corner point map and character content description information of each stele image; the corner point map is an image in which the corner points of all characters in the stele image are marked;
[0142] The feature extraction module 202 obtains shallow features of each stele image according to the stele image and the corner map of the stele image, and extracts deep features from the shallow image features to obtain deep features of the stele image;
[0143] The prediction module 203 predicts each deep feature, obtains the predicted character of the stone tablet image corresponding to the deep feature, and obtains the image prior knowledge of the predicted character; the image prior knowledge is used to describe the image information generated according to the predicted character;
[0144] Fusion module 204, using a fusion model to fuse the shallow features, image prior knowledge, and character content description information corresponding to each stele image to obtain the final features of each stele image;
[0145] The first reconstruction module 205 reconstructs each final feature using the reconstruction model to obtain a reconstructed image of the stone tablet image corresponding to each final feature;
[0146] The optimization module 206 constructs a reconstruction loss function and a pixel loss function based on all the reconstructed images, constructs a boundary loss function based on all the final features, and optimizes the fusion model and the reconstruction model using the reconstruction loss function, the pixel loss function and the boundary loss function to obtain an optimized fusion model and an optimized reconstruction model; the reconstruction loss function and the pixel loss function are used to describe the accuracy of all the reconstructed images, and the boundary loss function is used to describe the accuracy of all the final features;
[0147] The second reconstruction module 207 reconstructs the to-be-reconstructed stele image using the optimized fusion model and the optimized reconstruction model to obtain a final reconstructed image of the to-be-reconstructed stele image.
[0148] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0149] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0150] like Figure 3 As shown, an embodiment of the present application provides a terminal device. The terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 3 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above-mentioned method embodiments when executing the computer program D102.
[0151] Specifically, when the processor D100 executes the computer program D102, it obtains a plurality of stele images and extracts a corner map and character content description information of each stele image, then obtains shallow features of the stele image based on the stele image and the corner map of the stele image for each stele image, performs deep feature extraction on the shallow image features to obtain deep features of the stele image, predicts the deep features for each deep feature to obtain predicted characters of the stele image corresponding to the deep features, obtains image prior knowledge of the predicted characters, and then uses a fusion model to fusion the shallow features, image prior knowledge, and character content description information of each stele image. The character content description information is fused to obtain the final features of each stele image, and then each final feature is reconstructed using a reconstruction model to obtain a reconstructed image of the stele image corresponding to each final feature, and then a reconstruction loss function and a pixel loss function are constructed based on all reconstructed images, and a boundary loss function is constructed based on all final features, and the fusion model and the reconstruction model are optimized using the reconstruction loss function, the pixel loss function and the boundary loss function to obtain an optimized fusion model and an optimized reconstruction model, and finally the optimized fusion model and the optimized reconstruction model are used to reconstruct the stele image to be reconstructed to obtain the final reconstructed image of the stele image to be reconstructed. Among them, obtaining a corner point map of the stele image can provide corner point information of the stele image, so that the information richness of the deep features obtained based on the corner point information is improved, thereby improving the accuracy of the acquired image prior knowledge, and the accuracy and information richness of the final features obtained by fusion of accurate image prior knowledge, character content description information, and shallow features are improved, and the accuracy of the reconstructed image obtained by reconstruction based on the final features with high accuracy and richness is improved.
[0152] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0153] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device D10. Further, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.
[0154] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0155] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the digital reconstruction method device / terminal device of the stele image, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0157] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0158] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0159] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for digital reconstruction of a stone tablet image, characterized in that: include: Acquire multiple stone tablet images, and extract corner point maps and character content description information of each stone tablet image; The corner point map is an image in which the corner points of all characters in the stele image are marked; For each of the stele images, shallow features of the stele image are obtained according to the stele image and the corner point map of the stele image, and deep features are extracted from the shallow image features to obtain deep features of the stele image; For each of the deep features, the deep features are predicted to obtain predicted characters of the stone tablet image corresponding to the deep features, and image prior knowledge of the predicted characters is obtained; the image prior knowledge is used to describe image information generated according to the predicted characters; Using a fusion model to fuse shallow features, image prior knowledge, and character content description information corresponding to each of the stele images, to obtain final features of each of the stele images; Reconstructing each of the final features using a reconstruction model to obtain a reconstructed image of the stone tablet image corresponding to each of the final features; A reconstruction loss function and a pixel loss function are constructed based on all reconstructed images, a boundary loss function is constructed based on all final features, and the fusion model and the reconstruction model are optimized using the reconstruction loss function, the pixel loss function and the boundary loss function to obtain an optimized fusion model and an optimized reconstruction model; the reconstruction loss function and the pixel loss function are used to describe the accuracy of all reconstructed images, and the boundary loss function is used to describe the accuracy of all final features; The stone tablet image to be reconstructed is reconstructed using the optimized fusion model and the optimized reconstruction model to obtain a final reconstructed image of the stone tablet image to be reconstructed.
2. The digital reconstruction method according to claim 1, characterized in that: The obtaining of shallow features of the stone tablet image according to the stone tablet image and the corner point map of the stone tablet image comprises: By formula: R m,2 ,C2=UNet(R m,1 ,C1) R m,1 ,C1=ResNet(R m,0 ,C0) R m,0 ,C0=Init(i stone,m ,Cor stone,m ) Get the shallow feature R of the mth stele image m,2 ; Among them, C2, C1, and C0 all represent the number of feature map channels, UNet() represents the Unet network operation, and R m,1 Represents secondary features, ResNet() represents ResNet network operation, R m,0 Indicates the initial features, Init() indicates the convolution layer and pooling layer operations, I stone,m represents the mth stele image, Cor stone,m represents the corner point map of the mth stele image, m=1, 2, ..., M, and M represents the total number of stele images.
3. The digital reconstruction method according to claim 1, characterized in that: The fusion model is used to fuse the shallow features, image prior knowledge, and character content description information corresponding to each of the stele images to obtain the final features of each of the stele images, including: For each of the stele images, perform the following steps: The shallow features and the corner map corresponding to the stele image are fused to obtain an initial fused feature, and the initial fused feature is fused with the image prior knowledge corresponding to the stele image to obtain a secondary fused feature; Feature extraction is performed on the character content description information corresponding to the stele image to obtain text features, and the text features are fused with the secondary fusion features to obtain final features of the stele image.
4. The digital reconstruction method according to claim 1, characterized in that: The step of reconstructing each of the final features using the reconstruction model to obtain a reconstructed image of the stone tablet image corresponding to each of the final features includes: By formula: R m,SR =Pix(R m,2 +RL m,cross ) Get the reconstructed image R of the mth stele image m,SR ; Among them, Pix() represents upsampling, R m,2 represents the shallow features of the m-th stele image, RL m,cross represents the final feature of the mth stele image, m=1, 2, ..., M, and M represents the total number of stele images.
5. The digital reconstruction method according to claim 4, characterized in that: The reconstruction loss function and the pixel loss function are constructed based on all the reconstructed images, including: Performing text recognition on each of the reconstructed images to obtain a text recognition result for each of the reconstructed images, and constructing a reconstruction loss function based on all the text recognition results; A pixel loss function is constructed based on all reconstructed images.
6. The digital reconstruction method according to claim 5, characterized in that: The reconstruction loss function is: the rec =1 / m∑(before sr,m -pre hr,m ) 2 Among them, l rec Represents the value of the reconstruction loss function, pre sr,m represents the text recognition result of the mth reconstructed image, pre hr,m represents the true text label of the m-th stele image; The pixel loss function is: Among them, l2 represents the value of the pixel loss function, represents the pixel value of the i-th pixel point of the m-th reconstructed image, Represents the pixel value of the i-th pixel point of the m-th stone tablet image.
7. The digital reconstruction method according to claim 1, characterized in that: The boundary loss function is: Among them, l box represents the value of the boundary loss function, t m represents the bounding box parameters obtained according to the final features corresponding to the m-th stele image, represents the true bounding box parameters of the m-th stele image.
8. A digital reconstruction device for a stone tablet image, characterized in that: include: An acquisition module, which acquires a plurality of stele images and extracts a corner point map and character content description information of each stele image; The corner point map is an image in which the corner points of all characters in the stele image are marked; A feature extraction module, for each of the stele images, obtains shallow features of the stele image according to the stele image and the corner point map of the stele image, and performs deep feature extraction on the shallow image features to obtain deep features of the stele image; A prediction module predicts each of the deep features, obtains the predicted characters of the stone tablet image corresponding to the deep features, and acquires image prior knowledge of the predicted characters; the image prior knowledge is used to describe image information generated according to the predicted characters; A fusion module, which uses a fusion model to fuse the shallow features, image prior knowledge, and character content description information corresponding to each of the stone tablet images to obtain the final features of each of the stone tablet images; A first reconstruction module reconstructs each of the final features using a reconstruction model to obtain a reconstructed image of the stone tablet image corresponding to each of the final features; An optimization module, constructing a reconstruction loss function and a pixel loss function based on all reconstructed images, constructing a boundary loss function based on all final features, and optimizing the fusion model and the reconstruction model using the reconstruction loss function, the pixel loss function and the boundary loss function to obtain an optimized fusion model and an optimized reconstruction model; the reconstruction loss function and the pixel loss function are used to describe the accuracy of all reconstructed images, and the boundary loss function is used to describe the accuracy of all final features; The second reconstruction module reconstructs the to-be-reconstructed stone tablet image by using the optimized fusion model and the optimized reconstruction model to obtain a final reconstructed image of the to-be-reconstructed stone tablet image.
9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for digitally reconstructing a stone tablet image as described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for digitally reconstructing a stone tablet image as described in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Method for extracting stele inscription digital rubbing based on three-dimensional data scanning
CN104268924A
Scene character visual question-answering method based on knowledge-guided deep attention network
CN113792177A
Remote sensing image reconstruction method and device, processor and electronic equipment
CN114881860A
Construction method, device and application of image super-resolution reconstruction model
CN116993592A
License plate corner detection method and device and terminal equipment
CN116994233A
