Image processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides an image processing method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of image processing. The method comprises the following steps: performing feature extraction on an original image of a target object to obtain a feature map; performing bidirectional stripe convolution operation on the feature map based on the segmentation mask of the feature map, and extracting original edge features from the feature map; classifying the original edge features in a comparative learning mode to obtain classified feature information; and generating an edge feature enhanced image of the target object based on the classification feature information and the original image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, specifically to the field of image acquisition and processing technology, and in particular to an image processing method, device, electronic device, storage medium and program product. Background Art

[0002] As a means of human-computer interaction, voice interaction technology has been widely used in various small-screen smart devices, such as smartphones, tablets, smart watches, and various smart wearable devices that are not convenient for touch operation.

[0003] With the increasing popularity of the concept of smart home and people's yearning for an increasingly better life, various large-screen devices in homes or indoors are gradually combined with intelligence to form smart large-screen devices.

[0004] How to combine the voice interaction technology originally adapted to small-screen devices with large-screen devices to provide users with a better user experience is a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] The embodiments of the present disclosure provide an image processing method, an apparatus, an electronic device, a storage medium, and a program product.

[0006] In a first aspect, an embodiment of the present disclosure proposes an image processing method, comprising: performing feature extraction on an original image of a target object to obtain a feature map; performing a bidirectional strip convolution operation on the feature map based on a segmentation mask of the feature map to extract original edge features from the feature map; classifying the original edge features by contrastive learning to obtain classification feature information; and generating an edge feature enhanced image of the target object based on the classification feature information and the original image.

[0007] In the second aspect, an embodiment of the present disclosure proposes an image processing device, comprising: a feature map extraction module, configured to extract features from an original image of a target object to obtain a feature map; an edge feature extraction module, configured to perform a bidirectional strip convolution operation on the feature map based on a segmentation mask of the feature map, and extract original edge features from the feature map; a classification feature generation module, configured to classify the original edge features by contrastive learning to obtain classification feature information; and an enhanced image generation module, configured to generate an edge feature enhanced image of the target object based on the classification feature information and the original image.

[0008] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the image processing method described in any implementation method in the first aspect when executing the instructions.

[0009] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the image processing method described in any implementation manner in the first aspect when executed.

[0010] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the image processing method described in any implementation manner in the first aspect.

[0011] The disclosed embodiment, based on the original image of the target object, performs enhancement processing on the original image through a series of processing processes such as feature extraction, bidirectional strip convolution, and feature classification, so that the enhanced image after the enhancement processing not only maintains a complete display of the overall structure of the target object, but also further refines the detail features that may appear in the target object, so that these detail features are more clearly visible, and the scope of the target objects targeted by the image processing is increased, and some target objects containing subtle texture features and line features can be displayed more clearly.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0014] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0015] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0016] Figure 3 A flowchart of another image processing method provided by an embodiment of the present disclosure;

[0017] Figure 4 A flowchart of another image processing method provided by an embodiment of the present disclosure;

[0018] Figure 5 A flowchart of another image processing method provided by an embodiment of the present disclosure;

[0019] Figure 6 A schematic diagram of a flow chart of an image processing method in an application scenario provided by an embodiment of the present disclosure;

[0020] Figure 7 A structural block diagram of an image processing device provided by an embodiment of the present disclosure;

[0021] Figure 8 A schematic diagram of the structure of an electronic device suitable for executing an image processing method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0023] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0024] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the image processing method, apparatus, electronic device, and computer-readable storage medium of the present disclosure can be applied.

[0025] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0026] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 and server 105 may be installed with various applications for realizing information communication between the two, such as instant messaging applications.

[0027] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0028] The server 105 can provide various services through various built-in applications. It should be noted that the acquired original images can be obtained from the terminal devices 101, 102, 103 through the network 104, or can be pre-stored locally in the server 105 in various ways. Therefore, when the server 105 detects that the data has been stored locally, it can choose to directly obtain the data locally. In this case, the exemplary system architecture 100 may also not include the terminal devices 101, 102, 103 and the network 104.

[0029] Since the optimization process based on the original image may require more computing resources and stronger computing power, the image processing methods provided in the subsequent embodiments of the present disclosure are generally performed by the server 105 with stronger computing power and more computing resources, and accordingly, the image processing device is generally also set in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the above-mentioned various operations that are handed over to the server 105 through the image processing applications installed thereon, and then output the same results as the server 105. Especially in the case where there are multiple terminal devices with different computing powers at the same time, but when the image processing application determines that the terminal device has stronger computing power and more remaining computing resources, the terminal device can be allowed to perform the above-mentioned operations, thereby appropriately reducing the computing pressure of the server 105. Accordingly, the image processing device can also be set in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104.

[0030] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0031] Please refer to Figure 2 , Figure 2 This is a flowchart of an image processing method provided by an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0032] Step 201: extract features from the original image of the target object to obtain a feature map.

[0033] This step is intended to be performed by the execution subject of the image processing method (e.g. Figure 1 The server 105 shown in the figure obtains the original image of the target object, and performs feature extraction on the original image to obtain a feature map. Among them, the target object refers to an object whose image can be obtained by taking a photo or other means, such as a person, a document, a book, a cultural relic, and other items. The original image of the target object can be a photo obtained by taking a photo with a camera or other photographic device, etc. The execution subject can be to control the camera to complete the photo taking, thereby obtaining the original image of the target object, or to obtain pre-stored photos from the cloud or other servers, etc., and the present embodiment is not limited to this. Feature extraction refers to extracting information or features that are useful for tasks (such as classification, recognition, detection, etc.) from the original data. In the field of image processing, feature extraction generally refers to extracting descriptors or feature vectors that can represent the content of the original image to form a feature map.

[0034] Step 202: Perform a bidirectional strip convolution operation on the feature map based on the segmentation mask of the feature map to extract the original edge features from the feature map.

[0035] This step is intended to extract the original edge features from the feature map by the above-mentioned execution subject through a bidirectional strip convolution operation. In this embodiment, the execution subject can classify each pixel or region according to the information in the feature map through a classifier, assign a category label to each pixel or region according to the classification result, and combine to generate a segmentation mask. Then a bidirectional strip convolution operation is performed based on the segmentation mask. Among them, the strip convolution operation is a convolution operation in which the convolution kernel is strip-shaped. The size of the convolution kernel of the strip convolution in a certain dimension (usually width or height) is much smaller than another dimension, which enables it to better capture the strip features in the data. In scenes such as image segmentation and target detection, the strip convolution can better capture the slender objects or structures in the image (such as slender objects in the image, character sequences in the text, etc.). Bidirectional strip convolution refers to a strip convolution operation that considers both forward information and backward information, so that more comprehensive feature information can be extracted to obtain the original edge features. In practical applications, the above process can be used to highlight the texture features of one or some specific areas of the target object. Exemplarily, the execution entity may also generate an adaptive attention weight map based on the obtained original edge features, and adjust the attention weights for different areas and different features in the original image through the attention weight map, so as to characterize and distinguish the areas or features that need to be focused on.

[0036] Step 203: classify the original edge features by contrastive learning to obtain classification feature information.

[0037] This step is intended to enable the above-mentioned execution subject to classify the original edge features through contrastive learning to obtain classification feature information. Contrastive learning is a self-supervised learning method that learns the representation of data by comparing positive samples (for example, samples that are similar but different after data enhancement or transformation) and negative samples (for example, samples that are different from positive samples), so that positive samples are closer in the representation space and negative samples are farther away. This representation learning method can extract key features from the data and provide strong support for subsequent classification tasks. In this embodiment, the original edge features can be classified by contrastive learning to obtain classification feature information.

[0038] Step 204: Generate an edge feature enhanced image of the target object based on the classification feature information and the original image.

[0039] This step is intended to be performed by the above-mentioned execution subject, after extracting the classification feature information from the original image, combining the classification feature information with the original image to generate an edge feature enhanced image of the target object. Through this process, it is possible to achieve the overall structure of the target object presented in the edge feature enhanced image while highlighting the local detail features (i.e., the features corresponding to the edge features).

[0040] The image processing method provided by the embodiment of the present disclosure, based on the original image of the target object, enhances the original image through a series of processing processes such as feature extraction, bidirectional strip convolution, feature classification, etc., so that the enhanced image after the enhancement processing not only maintains a complete display of the overall structure of the target object, but also further refines the detail features that may appear in the target object, so that these detail features are more clearly visible, and the scope of the target objects targeted by the image processing is increased, and some target objects containing subtle texture features and line features can be displayed more clearly.

[0041] In some optional implementations of the disclosed embodiments, the above step 203, the process of classifying the original edge features by contrastive learning to obtain the classification feature information, mainly includes: first, extracting the same type of edge features and different type of edge features in the edge feature information; then, adjusting the classification feature information by reducing the feature distance of the same type of edge features and / or expanding the feature distance between the same type of edge features and different type of edge features. Exemplarily, the process of implementing the classification by contrastive learning can first perform data enhancement, and common data enhancement techniques include cropping, flipping, rotation, random cropping and color transformation. By generating different instances, the variability of the data is increased. Then the encoder network is trained, and the encoder network takes the data enhancement instances as input and maps them to the potential representation space, in which meaningful features and similarities are captured. The encoder network is usually a deep neural network architecture, such as a convolutional neural network (CNN) for image data or a recurrent neural network (RNN) for sequential data. The network learns to extract and encode high-level representations from the data enhancement instances, thereby facilitating the distinction between similar and dissimilar instances in subsequent steps. A projection network can then be used to take the output of the encoder network and project it into a lower dimensional space, often called a projection space or embedding space. This additional projection step helps to enhance the discriminative power of the learned representation. By mapping the representation to a lower dimensional space, the projection network reduces the complexity and redundancy of the data, helping to better separate similar and dissimilar instances. After the enhanced instances are encoded and projected into the embedding space, contrastive learning is applied, aiming to maximize the consistency between positive pairs (instances from the same sample) and minimize the consistency between negative pairs (instances from different samples). This brings the feature distances of similar edge features closer together, and increases the feature distances between similar edge features and non-similar edge features, thereby achieving classification and obtaining classification feature information.

[0042] It should be noted that the above process is only an example. In a specific implementation, data enhancement or projection network steps may not be performed as needed. In addition, for the distance change between the same type of edge features and the different type of edge features, only one of them may be adjusted according to the actual situation, and this embodiment is not limited to this.

[0043] In the embodiment of the present disclosure, the above process is a process of enhancing the original image through a series of processing processes such as feature extraction, bidirectional strip convolution, and feature classification. One of the main functions of this process is to focus on enhancing texture features. In practical applications, in addition to enhancing texture features, other features in the target object can also be enhanced. In some optional implementations of the embodiment of the present disclosure, such as Figure 3 As shown, the image processing method also includes:

[0044] Step 301: Based on a pre-trained image processing model, surface feature recognition is performed on the original image to obtain a feature recognition result.

[0045] In this embodiment, the training process of the image processing model mainly includes the following steps:

[0046] Step 1: Extract high-level features and low-level features of the sample image through the initial neural network model. In the fields of image processing, computer vision, and deep learning, high-level features, also known as high-level semantic features, usually refer to features extracted from the deeper layers of deep learning models (such as convolutional neural networks), which contain rich combination information. Low-level features, also known as underlying features, usually refer to features extracted from the shallower layers of deep learning models. These features are closer to the original pixel data of the image. Low-level features of images usually refer to contours, edges, colors, textures, and shape features. High-level features and low-level features have their own advantages and disadvantages in image understanding and analysis, and they are complementary. High-level features can provide the overall content and meaning of the image, while low-level features can provide local details of the image and accurate target location information.

[0047] Step 2: Calculate the difference vector based on high-level features and low-level features.

[0048] After extracting the high-level features and the low-level features, the execution body may calculate the difference vector based on the high-level features and the low-level features. For example, the obtained difference vector may be post-processed, such as normalization processing, smoothing processing, etc., to eliminate noise or improve the interpretability of the vector.

[0049] Step 3: Adjust the weight distribution of the neural network model based on the difference vector to obtain the image processing model.

[0050] In the process of training the initial neural network model, the difference vector can be used as a guide to calculate the gradient of the weight. The gradient can indicate how the weight should be adjusted to reduce the difference between the feature levels. In addition, an optimization algorithm (such as gradient descent, etc.) can be used to update the weights of the neural network model according to the calculated gradient. The goal of the weight update is to minimize the difference between high-level features and low-level features, thereby improving the performance of the model and ultimately obtaining the image processing model.

[0051] It should be noted that, in this embodiment, the process of performing surface feature recognition on the original image based on a pre-trained image processing model to obtain feature recognition results can also be achieved through a neural network model such as CNN or RNN, and this embodiment is not limited to this.

[0052] Step 302: Generate a first optimized image based on the feature recognition result.

[0053] This step aims to generate a first optimized image of the original image based on the feature recognition result by the execution subject. In this embodiment, through the above process, the surface features (such as bumps and undulations) in the original image of the target object detected can be more prominently detected by the pre-trained image processing model, which can not only highlight the main undulation changes on the surface of the target object, but also retain the subtle bump levels, so as to maintain the overall structure while not ignoring the local fine features.

[0054] Step 303: Integrate the edge feature enhanced image and the first optimized image to obtain a first integrated image.

[0055] In this embodiment, through the above Figure 2 The edge feature enhanced image obtained by steps 201-204 of the illustrated embodiment, and the first optimized image obtained by steps 301-302, can be further integrated to obtain a first integrated image. The first integrated image integrates the detail features such as the texture features highlighted by the edge feature enhanced image and the surface features highlighted by the first optimized image, and can more comprehensively and specifically reflect the features in the target object, so that the first integrated image can more clearly and closely display the target object. Among them, the integration process of the edge feature enhanced image and the first optimized image can be integrated through an attention mechanism. Exemplarily, the attention mechanism is first applied to generate an attention weight map (attention distribution), and the attention weight map is used to indicate which areas in the image are important for the integration task. Then, the extracted features are weighted using the attention weight map (attention distribution) to highlight the features of important areas and suppress the features of unimportant areas. The features from different input images are spliced ​​or weighted summed according to the attention weights to achieve feature integration, thereby obtaining the first integrated image.

[0056] It should be noted that, in this embodiment, the process of integrating different images is not limited to being implemented through the attention mechanism, but can also be implemented using more mature image fusion algorithms in related fields (such as Alpha fusion algorithm (alphacompositing), pyramid fusion algorithm, Poisson fusion algorithm, etc.), and this embodiment is not limited to this.

[0057] In practical applications, in addition to enhancing texture features, other features in the target object may also be enhanced. In some optional implementations of the embodiments of the present disclosure, such as Figure 4 As shown, the image processing method also includes:

[0058] Step 401: The original image is optimized by adaptive histogram equalization and multi-scale processing to obtain a second optimized image. Among them, adaptive histogram equalization is a computer image processing technology for image enhancement, which aims to improve the contrast of the image. The image contrast can be changed by calculating the local histogram of the image and then redistributing the brightness. Multi-scale processing refers to the use of operations or representations of different scales to extract and analyze image information in image processing, which helps to capture multi-scale features in the image and improve the robustness and performance of image processing. In specific implementation, the execution subject can first perform multi-scale processing on the original image to extract feature information of different scales, and then perform adaptive histogram equalization on the image at each scale to enhance the local contrast of the image, and finally perform feature fusion to obtain the second optimized image. Through the above process, it is possible to capture the multi-scale features in the original image and enhance the local contrast of the original image at the same time, thereby improving the performance and effect of image processing.

[0059] Step 402: Integrate the edge feature enhanced image, the first optimized image and the second optimized image to obtain a second integrated image.

[0060] Further, in this embodiment, through the above Figure 2The edge feature enhanced image obtained by steps 201-204 of the illustrated embodiment, the first optimized image obtained by steps 301-302, and the second optimized image obtained by step 401 can be further integrated to obtain a second integrated image. The second integrated image integrates the detail features such as texture features highlighted by the edge feature enhanced image, the surface features highlighted by the first optimized image, and the multi-scale features and contrast features highlighted by the second optimized image, which can more comprehensively and specifically reflect the features in the target object, so that the second integrated image can more clearly and closely display the target object, not only with high computational efficiency, but also with the ability to automatically adjust the integration ratio according to the feature quality of different regions, and finally output a balanced and complete integrated image. Among them, the integration process of the edge feature enhanced image and the first optimized image and the second optimized image can be integrated through an attention mechanism. Exemplarily, the attention mechanism is first applied to generate an attention weight map (attention distribution), and the attention weight map is used to indicate which areas in the image are important for the integration task. Then, the extracted features are weighted using the attention weight map (attention distribution) to highlight the features of important areas and suppress the features of unimportant areas. The features from different input images are concatenated or weighted-summed according to the attention weights to achieve feature integration, thereby obtaining the second integrated image.

[0061] In some optional implementations of the disclosed embodiments, the execution subject may also integrate the edge feature enhanced image and the second optimized image described in the above embodiments as a whole to obtain a third integrated image. The third integrated image integrates the detail features such as texture features highlighted by the edge feature enhanced image and the multi-scale features and contrast features highlighted by the second optimized image, and can more comprehensively and specifically reflect the features in the target object, so that the third integrated image can more clearly and closely display the target object. Among them, the integration process of the edge feature enhanced image and the second optimized image can be integrated through an attention mechanism. Exemplarily, the attention mechanism is first applied to generate an attention weight map (attention distribution), and the attention weight map is used to indicate which areas in the image are important for the integration task. Then, the extracted features are weighted using the attention weight map (attention distribution) to highlight the features of important areas and suppress the features of unimportant areas. The features from different input images are spliced ​​or weighted summed according to the attention weights to achieve feature integration, thereby obtaining the third integrated image.

[0062] Please refer to Figure 5 , Figure 5 A flowchart of another image processing method provided by an embodiment of the present disclosure, wherein process 500 includes the following steps:

[0063] Step 501: Acquire a test image obtained by photographing a target object with a photographing device in any photographing posture.

[0064] In this embodiment, the process of acquiring the original image of the target object can be obtained by shooting with a shooting device. And, first, the target object can be shot with the shooting device in any shooting posture to obtain a test image. The shooting posture mainly refers to the position and angle of the shooting device during shooting. The execution subject can obtain the test image shot by the shooting device, as well as the shooting posture when shooting the test image.

[0065] Step 502: Determine the target shooting posture based on the actual shooting posture of the test image obtained by shooting.

[0066] Based on the actual shooting posture of the test image shot by the shooting device, the execution subject can determine the target shooting posture to be actually shot for the target object. For example, to achieve fixed 4-view or 8-view shooting, the contour information and size information of the target object can be determined according to the test image, and then combined with the contour information and size information, the specific shooting posture information (i.e., shooting angle and position, etc.) suitable for 4-view or 8-view shooting can be determined. For example, when the target object is a large object (such as cultural relics such as seismometers, detection equipment such as X-ray detectors, etc.), the specific shooting posture determined can be in a larger acquisition area, and the shooting angle can also span a larger span; and when the target object is a small object (such as cultural relics such as oracle bones engraved with oracle bone inscriptions, miniature sculptures, etc.), the specific shooting posture determined is in a smaller acquisition area, closer to the target alignment, and the span of the shooting angle is also smaller. The specific shooting posture can be adjusted according to the actual situation of the target object, and this embodiment is not limited to this.

[0067] Step 503: Acquire an original image obtained by photographing the target object by the photographing device in the target photographing posture.

[0068] After determining the target shooting posture for actual shooting, the shooting device can be used to perform actual shooting at the target shooting posture to obtain original images of the target object at different positions and angles, and the execution subject can obtain the original image.

[0069] Step 504: extract features from the original image of the target object to obtain a feature map.

[0070] Step 505: Perform a bidirectional strip convolution operation on the feature map based on the segmentation mask of the feature map to extract the original edge features from the feature map.

[0071] Step 506: Classify the original edge features by contrastive learning to obtain classification feature information.

[0072] Step 507: Generate an edge feature enhanced image of the target object based on the classification feature information and the original image.

[0073] The above steps 504-507 are similar to Figure 2 Steps 201-204 shown are consistent, and for the same contents, please refer to the corresponding parts of the previous embodiment, which will not be repeated here.

[0074] In specific implementation, the execution subject may also integrate the edge feature images corresponding to the original images collected at different shooting positions according to the target shooting posture. For example, for 4-view shooting (including horizontal angle, downward angle, upward angle, and side angle), the original images of the target object at different angles can be obtained, and the corresponding edge feature enhanced images can be obtained through the above steps 504-507. The execution subject can integrate the edge feature enhanced images of the target object according to different angles in combination with the shooting postures corresponding to the 4-view shooting.

[0075] Through the above process, on the one hand, automatic adjustment of shooting for target objects of different sizes and shapes can be achieved to improve shooting efficiency, and on the other hand, all aspects of characteristic information of the target object can be reflected more comprehensively, accurately and clearly, so that the information of the target object presented in the obtained image is more in line with the actual situation of the target object.

[0076] The image processing method described in any of the above embodiments, the generated enhanced image, integrated image, etc., are all processing processes for generating images for the target object. In practical applications, for the target object, in order to better present the target object, a three-dimensional modeling process may be further performed based on the enhanced image or integrated image obtained in any of the above embodiments to generate a three-dimensional model of the target object. In some optional implementations of the embodiments of the present disclosure, the above execution subject may also perform the following steps:

[0077] Step 508: Perform three-dimensional modeling on the target object based on the edge feature enhanced image and the target shooting posture to obtain a three-dimensional model of the target object.

[0078] In this embodiment, the process of the execution subject performing three-dimensional modeling of the target object based on the edge feature enhanced image and the target shooting posture may include multiple situations: the execution subject performs three-dimensional modeling directly based on the edge feature enhanced image and the target shooting posture, the execution subject performs three-dimensional modeling based on the first integrated image obtained by integrating the edge feature enhanced image and the first optimized image and the target shooting posture, the execution subject performs three-dimensional modeling based on the second integrated image obtained by integrating the edge feature enhanced image and the second optimized image and the target shooting posture, and the execution subject performs three-dimensional modeling based on the third integrated image obtained by integrating the edge feature enhanced image with the first optimized image and the second optimized image, and the target shooting posture. The specific method used for three-dimensional modeling can be selected and configured according to actual needs, and this embodiment is not limited to this.

[0079] In some implementations of the embodiments of the present disclosure, step 508, the process of performing three-dimensional modeling of the target object based on the edge feature enhanced image and the target shooting posture to obtain the three-dimensional model of the target object mainly includes:

[0080] Step 1: Perform group feature extraction on the edge feature enhanced image based on the target shooting posture to obtain modeling feature information. The target shooting posture includes the shooting position and posture, which determines the content and viewing angle of the original image taken, and will affect the accuracy and authenticity of the three-dimensional modeling. Therefore, the execution subject obtains the original image of the target object at different viewing angles based on the target shooting posture, and performs corresponding processing to obtain the edge feature enhanced image, and performs group feature extraction on the edge feature enhanced image (for example, group feature extraction is performed according to grid division) to obtain modeling feature information.

[0081] Step 2: Perform point cloud sampling based on modeling feature information to obtain point cloud data.

[0082] After obtaining the modeling feature information, the execution subject can perform point cloud sampling based on the modeling feature information to obtain point cloud data. For example, for a target area with more features in the target object, the sampling density of point cloud sampling can be increased, and sampling can be performed at a first sampling density; for a non-target area with fewer features (i.e., an area other than the target area), the sampling density of point cloud sampling can be reduced, and sampling can be performed at a second sampling density, where the first sampling density is greater than the second sampling density. In this way, more data can be concentrated in the target area, and while ensuring the sampling of the detailed features of the target object, processing resources can be reasonably allocated to improve processing efficiency.

[0083] Step 3: Perform mesh reconstruction and texture mapping based on point cloud data to obtain a three-dimensional model.

[0084] After acquiring the point cloud data through point cloud sampling, the execution subject can obtain a three-dimensional model of the target object through mesh reconstruction and texture mapping processing.

[0085] The image processing method of the disclosed embodiment performs three-dimensional modeling based on the enhancement processing of the original image of the target object. On the one hand, it can more accurately and clearly identify and display the detail features, texture features, etc. on the target object, and on the other hand, it can further build these features into the three-dimensional model of the target object, and can more closely and comprehensively display the content on the target object, so that the display quality of the image and the model has been greatly improved, and the modeling quality of small objects (such as oracle bones engraved with oracle bone inscriptions) with rich detail features on the surface has been significantly improved, providing more reliable three-dimensional data support for subsequent digital research and maintenance work.

[0086] To deepen understanding, the present disclosure also provides a specific implementation solution in combination with a specific application scenario, see Figure 6 Flow 600 is shown.

[0087] In this embodiment, the target object is a small cultural relic (such as an oracle bone with oracle bone inscriptions) as an example for description.

[0088] The process 600 mainly includes:

[0089] Step 601: Acquire a test image 3 of an oracle bone 2 taken by a photographing device 1 .

[0090] Step 602: Determine the target shooting posture according to the actual shooting posture of the oracle bone 2 shot by the trial image 3.

[0091] Step 603: Acquire the original image 8 captured by the photographing device 1 at the target photographing position (photographing position 4-7).

[0092] Step 604: extract features from the original image 8 to obtain a feature map.

[0093] Step 605: Perform a bidirectional strip convolution operation on the feature map based on the segmentation mask of the feature map to extract the original edge features.

[0094] Step 606: Classify the original edge features through comparative learning to obtain classification feature information.

[0095] Step 607: Generate an edge feature enhanced image 9 of the oracle bone 2 based on the classification feature information and the original image 8.

[0096] Step 608 : Perform three-dimensional modeling of the oracle bone 2 based on the edge feature enhanced image 9 and the target shooting posture to obtain a three-dimensional model 10 of the oracle bone 2 .

[0097] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an image processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0098] like Figure 7 As shown, the image processing device 700 of this embodiment may include: a feature map extraction module 701, an edge feature extraction module 702, a classification feature generation module 703 and an enhanced image generation module 704. Among them, the feature map extraction module 701 is configured to extract features from the original image of the target object to obtain a feature map. The edge feature extraction module 702 is configured to perform a bidirectional strip convolution operation on the feature map based on the segmentation mask of the feature map to extract the original edge features from the feature map. The classification feature generation module 703 is configured to classify the original edge features by contrastive learning to obtain classification feature information. The enhanced image generation module 704 is configured to generate an edge feature enhanced image of the target object based on the classification feature information and the original image.

[0099] In this embodiment, in the image processing device 700, the specific processing of the feature map extraction module 701, the edge feature extraction module 702, the classification feature generation module 703 and the enhanced image generation module 704 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0100] In some optional implementations of this embodiment, the classification feature generation module 703 includes: a feature extraction submodule configured to extract similar edge features and non-similar edge features in edge feature information. The classification feature generation submodule is configured to adjust the classification feature information in a manner of reducing the feature distance of similar edge features and / or expanding the feature distance between similar edge features and non-similar edge features.

[0101] In some optional implementations of this embodiment, the image processing device further includes: a feature recognition module, configured to perform surface feature recognition on the original image based on a pre-trained image processing model to obtain a feature recognition result. A first optimized image generation module, configured to generate a first optimized image based on the feature recognition result. The training process of the image processing model includes the following steps: extracting high-level features and low-level features of the sample image respectively through an initial neural network model. Calculating a difference vector based on the high-level features and the low-level features. Adjusting the weight distribution of the neural network model based on the difference vector to obtain an image processing model.

[0102] In some optional implementations of this embodiment, the image processing device further includes: a first integrated image generation module configured to integrate the edge feature enhanced image and the first optimized image to obtain a first integrated image.

[0103] In some optional implementations of this embodiment, the image processing device further includes: a second optimized image generation module, configured to optimize the original image through adaptive histogram equalization and multi-scale processing to obtain a second optimized image.

[0104] In some optional implementations of this embodiment, the image processing device further includes: a second integrated image generation module, configured to integrate the edge feature enhanced image, the first optimized image, and the second optimized image to obtain a second integrated image. The second integrated image generation module is further configured to: fuse the edge feature enhanced image, the first optimized image, and the second optimized image based on their respective corresponding attention distributions to obtain the second integrated image.

[0105] In some optional implementations of this embodiment, the image processing device further includes: a third integrated image generation module configured to integrate the edge feature enhanced image and the second optimized image to obtain a third integrated image.

[0106] In some optional implementations of this embodiment, the image processing device further includes: a test image acquisition module, configured to acquire a test image obtained by the shooting device at any shooting posture of the target object. A shooting posture determination module, configured to determine the target shooting posture based on the actual shooting posture of the test image. An original image acquisition module, configured to acquire an original image obtained by the shooting device at the target shooting posture of the target object.

[0107] The shooting posture determination module includes: a contour and size extraction module configured to extract contour information and size information of the target object based on the test image and the actual shooting posture; and a shooting posture determination submodule configured to determine the target shooting posture information based on the contour information and size information.

[0108] In some optional implementations of this embodiment, the image processing device further includes: an image integration module configured to integrate edge feature enhanced images corresponding to original images captured at different shooting positions based on the target shooting posture.

[0109] In some optional implementations of this embodiment, the image processing device also includes: a three-dimensional modeling module, which is configured to perform three-dimensional modeling on the target object based on the edge feature enhanced image and the target shooting posture to obtain a three-dimensional model of the target object.

[0110] In some optional implementations of this embodiment, the three-dimensional modeling module includes: a modeling feature extraction submodule, which is configured to perform group feature extraction on the edge feature enhanced image based on the target shooting posture to obtain modeling feature information. A point cloud data acquisition submodule, which is configured to perform point cloud sampling based on the modeling feature information to obtain point cloud data. The three-dimensional modeling submodule is configured to perform mesh reconstruction and texture mapping based on the point cloud data to obtain a three-dimensional model. Among them, the point cloud data acquisition submodule is further configured to: based on the modeling feature information and the shooting posture information, collect the first point cloud data of the target area with a first sampling density, and collect the second point cloud data outside the target area with a second sampling density to obtain point cloud data. Among them, the first sampling density is greater than the second sampling density.

[0111] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The image processing apparatus provided by this embodiment, on the basis of acquiring the original image of the target object, performs enhancement processing on the original image through a series of processing processes such as feature extraction, bidirectional strip convolution, and feature classification, so that the enhanced image after the enhancement processing not only maintains a complete display of the overall structure of the target object, but also further refines the detail features that may appear in the target object, so that these detail features are more clearly visible, and the scope of the target objects targeted by the image processing is increased, so that some target objects containing subtle texture features and line features can be displayed more clearly.

[0112] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the image processing method described in any of the above embodiments can be implemented when the at least one processor executes.

[0113] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the image processing method described in any of the above embodiments when executed.

[0114] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which can implement the image processing method described in any of the above embodiments when executed by a processor.

[0115] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0116] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0117] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0118] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the image processing method in any other appropriate manner (e.g., by means of firmware).

[0119] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0121] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0124] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0125] According to the technical solution of the embodiment of the present disclosure, on the basis of obtaining the original image of the target object, the original image is enhanced through a series of processing processes such as feature extraction, bidirectional strip convolution, and feature classification, so that the enhanced image after the enhancement processing not only maintains a complete display of the overall structure of the target object, but also further refines the detail features that may appear in the target object, so that these detail features are more clearly visible, and the scope of the target objects targeted by the image processing is increased, so that some target objects containing subtle texture features and line features can be displayed more clearly.

[0126] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0127] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: Extract features from the original image of the target object to obtain a feature map; Performing a bidirectional strip convolution operation on the feature map based on the segmentation mask of the feature map to extract original edge features from the feature map; Classifying the original edge features by contrastive learning to obtain classification feature information; An edge feature enhanced image of the target object is generated based on the classification feature information and the original image.

2. The method according to claim 1, wherein: The original edge features are classified by contrastive learning to obtain classification feature information, including: Extracting similar edge features and non-similar edge features from the edge feature information; The classification feature information is obtained by adjusting in a manner of reducing the feature distance of the same type of edge features and / or increasing the feature distance between the same type of edge features and the non-same type of edge features.

3. The method according to claim 1, further comprising: Based on a pre-trained image processing model, surface feature recognition is performed on the original image to obtain a feature recognition result; A first optimized image is generated based on the feature recognition result.

4. The method according to claim 3, wherein: The training process of the image processing model includes the following steps: The high-level features and low-level features of the sample image are extracted respectively through the initial neural network model; Calculating a difference vector based on the high-level features and the low-level features; The weight distribution of the neural network model is adjusted based on the difference vector to obtain the image processing model.

5. The method according to claim 3, further comprising: The edge feature enhanced image and the first optimized image are integrated to obtain a first integrated image.

6. The method according to claim 3, further comprising: The original image is optimized by adaptive histogram equalization and multi-scale processing to obtain a second optimized image.

7. The method according to claim 6, further comprising: Based on the attention distribution corresponding to the edge feature enhanced image, the first optimized image and the second optimized image, they are fused to obtain a second integrated image.

8. The method according to claim 1, further comprising: Acquire a test image of the target object captured by a shooting device in any shooting posture; Extracting contour information and size information of the target object based on the test image and the actual shooting posture; Determine the target shooting posture based on the contour information and size information; The original image obtained by the shooting device shooting the target object in the target shooting posture is obtained.

9. The method according to claim 8, further comprising: The target object is three-dimensionally modeled based on the edge feature enhanced image and the target shooting posture to obtain a three-dimensional model of the target object.

10. The method according to claim 9, wherein: The three-dimensional modeling is performed based on the edge feature enhanced image and the target shooting posture to obtain the three-dimensional model of the target object, including: Based on the target shooting posture, the edge feature enhanced image is subjected to group feature extraction to obtain modeling feature information; Performing point cloud sampling based on the modeling feature information to obtain point cloud data; Mesh reconstruction and texture mapping are performed based on the point cloud data to obtain the three-dimensional model.

11. The method according to claim 10, wherein: The step of performing point cloud sampling based on the modeling feature information to obtain point cloud data includes: Based on the modeling feature information and the shooting posture information, first point cloud data of the target area is collected with a first sampling density, and second point cloud data outside the target area is collected with a second sampling density to obtain the point cloud data; wherein the first sampling density is greater than the second sampling density.

12. An image processing device, comprising: A feature map extraction module is configured to extract features from an original image of a target object to obtain a feature map; an edge feature extraction module, configured to perform a bidirectional strip convolution operation on the feature map based on a segmentation mask of the feature map, and extract original edge features from the feature map; A classification feature generation module is configured to classify the original edge features by contrastive learning to obtain classification feature information; The enhanced image generation module is configured to generate an edge feature enhanced image of the target object based on the classification feature information and the original image.

13. The device according to claim 12, further comprising: A feature recognition module is configured to perform surface feature recognition on the original image based on a pre-trained image processing model to obtain a feature recognition result; The first optimized image generating module is configured to generate a first optimized image based on the feature recognition result.

14. The device according to claim 13, wherein: The training process of the image processing model includes the following steps: The high-level features and low-level features of the sample image are extracted respectively through the initial neural network model; Calculating a difference vector based on the high-level features and the low-level features; The weight distribution of the neural network model is adjusted based on the difference vector to obtain the image processing model.

15. The apparatus according to claim 13, further comprising: The first integrated image generation module is configured to integrate the edge feature enhanced image and the first optimized image to obtain a first integrated image.

16. The apparatus according to claim 13, further comprising: The second optimized image generation module is configured to optimize the original image through adaptive histogram equalization and multi-scale processing to obtain a second optimized image.

17. The apparatus according to claim 16, further comprising: The second integrated image generation module is configured to fuse the edge feature enhanced image, the first optimized image and the second optimized image based on their respective corresponding attention distributions to obtain a second integrated image.

18. The apparatus according to claim 12, further comprising: A test image acquisition module is configured to acquire a test image obtained by a shooting device shooting the target object in any shooting posture; A contour and size extraction module, configured to extract contour information and size information of the target object based on the test image and the actual shooting posture; A shooting posture determination module is configured to determine the target shooting posture based on the contour information and the size information; The original image acquisition module is configured to acquire the original image obtained by the shooting device shooting the target object in the target shooting posture.

19. The apparatus according to claim 18, further comprising: The three-dimensional modeling module is configured to perform three-dimensional modeling on the target object based on the edge feature enhanced image and the target shooting posture to obtain a three-dimensional model of the target object.

20. The device according to claim 19, wherein The three-dimensional modeling module includes: A modeling feature extraction submodule is configured to perform group feature extraction on the edge feature enhanced image based on the target shooting posture to obtain modeling feature information; A point cloud data acquisition submodule is configured to perform point cloud sampling based on the modeling feature information to obtain point cloud data; The three-dimensional modeling submodule is configured to perform mesh reconstruction and texture mapping based on the point cloud data to obtain the three-dimensional model.

21. The device according to claim 20, wherein: The point cloud data acquisition submodule is further configured to: Based on the modeling feature information and the shooting posture information, first point cloud data of the target area is collected with a first sampling density, and second point cloud data outside the target area is collected with a second sampling density to obtain the point cloud data; wherein the first sampling density is greater than the second sampling density.

22. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the image processing method according to any one of claims 1 to 11.

23. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the image processing method according to any one of claims 1 to 11.

24. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 11 are implemented.

Images