Image processing method and device, electronic equipment and storage medium

Abstract

The application relates to an image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining an original image to be processed; the original image is spliced based on at least two sub-images; target features are obtained by performing feature extraction on the original image; the target features comprise at least one of visual features, text semantic features and boundary features; the boundary position of each sub-image is determined by performing boundary detection on the original image through a boundary detection algorithm and combining the boundary features; the boundary position is verified based on the visual features and the text semantic features of the adjacent two side regions of the boundary position, and a verification result is obtained; in the case that the verification result is qualified, the original image is segmented based on the boundary position, and the sub-images are obtained. In this way, the segmentation boundary is judged from the content logic by fusing the visual features and the text semantic features, the key information in the long image content is reduced, and the integrity of the sub-image information is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] The long image integrates image and text information, can efficiently carry diversified contents such as commodity detail parameters, instant chat records, user experience single publishing and brand promotion posters, and is widely used in e-commerce product promotion, electronic manual compilation and social media content dissemination due to the advantages of high information density and intuitive visual expression.

[0003] The current mainstream long image division scheme is mainly constructed around image bottom layer visual features, extracts texture, color, edge and other features, combines threshold segmentation, region growing and edge detection and other classic algorithms, and divides the long image into several continuous visual regions to achieve the basic goal of sub-image division. However, such scheme still has significant technical limitations in actual application, and is difficult to adapt to high-precision division requirements in complex scenarios, for example, the existing scheme depends on fixed size cropping or simple rule matching (such as mechanical segmentation based on pixel rows), lacks the perception ability of long image content semantics, and is easy to cause sub-image content error cropping; seamless splicing design or complex background interference (such as gradient background, repeated texture) often exists in long image making, which makes it difficult for existing algorithms to accurately locate the real boundaries between sub-images.

[0004] Therefore, how to accurately and efficiently complete the division of the long image is a problem to be solved. SUMMARY

[0005] Therefore, it is necessary to provide an image processing method, device, electronic equipment and storage medium in view of the above technical problems.

[0006] In a first aspect, the present application provides an image processing method, which comprises:

[0007] Obtaining an original image to be processed; wherein the original image is spliced based on at least two sub-images;

[0008] Performing feature extraction on the original image to obtain target features; wherein the target features include at least one of visual features, text semantic features and boundary features;

[0009] Performing boundary detection on the original image by a boundary detection algorithm, and determining the boundary positions of each sub-image in combination with the boundary features;

[0010] Verifying the boundary positions based on the visual features and the text semantic features of the adjacent regions on both sides of the boundary positions to obtain a verification result;

[0011] In a case where the verification result is qualified, the original image is segmented based on the boundary position to obtain the sub-image.

[0012] In one of the embodiments, the method further comprises:

[0013] By performing integrity analysis on the boundary position of each sub-image, a candidate sub-image with mis-trim is determined to exist; wherein the integrity analysis comprises at least one of semantic integrity analysis and contour integrity analysis;

[0014] A repair content matching the text semantic feature of the adjacent area of the boundary position in the candidate sub-image is generated through a diffusion model, and the detail feature of the repair content is optimized through multiple rounds of iteration;

[0015] The optimized repair content is fused with the corresponding candidate sub-image to obtain a first target sub-image.

[0016] In one of the embodiments, the method further comprises:

[0017] A first image is screened from the sub-images, and sub-images other than the first image are determined to be content images; wherein the first image is an image used to provide a target image style migration;

[0018] Features of the first image and each of the content images are extracted respectively using a pre-trained convolutional neural network;

[0019] Statistical data are calculated based on each of the features; the statistical data include mean and standard deviation;

[0020] The features of the content images are subjected to instance normalization processing, and the statistical data of the normalized features are adjusted based on the statistical data corresponding to the first image to obtain processed features;

[0021] The processed features are input into an encoder to reconstruct a second target sub-image after style migration.

[0022] In one of the embodiments, the segmentation of the original image based on the boundary position to obtain the sub-image comprises:

[0023] A platform identifier of a platform to be published and a content type of content to be published are obtained;

[0024] The platform identifier and the content type are matched with a preset rule library to obtain a matching result; wherein the rule library includes a plurality of platform / content type corresponding trimming candidate rules;

[0025] According to the matched cutting candidate rule and the boundary position, the original image is segmented to obtain the sub-image.

[0026] In one of the embodiments, the method further comprises:

[0027] Obtaining a feedback index of the sub-image after being published to a platform to be published, wherein the feedback index comprises a click rate index, a satisfaction index and a stay time index;

[0028] Inputting the feedback index and the original image into a cutting decision model to obtain a segmentation optimization result, wherein the cutting decision model is used to indicate a mapping relationship between the feedback index and a segmentation strategy;

[0029] Segmenting the original image based on the segmentation optimization result and the boundary position to obtain a segmentation-optimized sub-image.

[0030] In one of the embodiments, the feature extraction of the original image to obtain a target feature comprises at least one of the following:

[0031] The first kind: using a CNN branch in a CNN-Transformer hybrid model to extract local texture features in the visual features of the original image, and using a Transformer branch in the CNN-Transformer hybrid model to extract context correlation features in the visual features;

[0032] The second kind: using optical character recognition to extract candidate text features from the original image, and inputting the candidate text features into a natural language processing model to obtain the text semantic features;

[0033] The third kind: using frequency domain analysis and / or Hough transform to analyze and extract the boundary features from the original image.

[0034] In one of the embodiments, the visual features comprise color features; and the verification of the boundary position based on the visual features and the text semantic features of the adjacent two-side regions of the boundary position obtains a verification result, which comprises:

[0035] Using a first branch network in a double-branch verification network to verify the color features of the adjacent two-side regions of the boundary position, and determining a first alternative verification result as qualified in a case that the color features of the adjacent two-side regions do not match;

[0036] The second branch network in the double-branch verification network verifies the text semantic features of the adjacent two side regions of the boundary position, and determines that the second candidate verification result is qualified in a case where semantic information represented by the text semantic features of the adjacent two side regions is inconsistent.

[0037] In a case where the first candidate verification result and the second candidate verification result are both qualified, the verification result is determined to be qualified.

[0038] In a second aspect, the present application further provides an image processing device, and the device comprises:

[0039] An acquisition module is configured to acquire an original image to be processed, wherein the original image is spliced based on at least two sub-images.

[0040] A feature extraction module is configured to perform feature extraction on the original image to obtain target features, wherein the target features comprise at least one of visual features, text semantic features, and boundary features.

[0041] A boundary detection module is configured to perform boundary detection on the original image by using a boundary detection algorithm, and determine boundary positions of the sub-images in combination with the boundary features.

[0042] A verification module is configured to verify the boundary positions based on the visual features and the text semantic features of the adjacent two side regions of the boundary positions, and obtain a verification result.

[0043] A segmentation module is configured to segment the original image based on the boundary positions to obtain the sub-images in a case where the verification result is qualified.

[0044] In a third aspect, the present application further provides an electronic device, which comprises a processor and a memory; wherein the memory is configured to store a computer program; and the processor is configured to execute the computer program to implement steps of a method performed by any of the embodiments of the present application.

[0045] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement steps of a method performed by any of the embodiments of the present application.

[0046] In the image processing method, on one hand, through the closed loop process of multi-feature extraction, accurate boundary positioning, bidirectional feature verification and segmentation under the condition of verification, the limitation of traditional fixed size or simple rule cropping can be broken through, the segmentation boundary is judged from the content logic by virtue of the fusion of visual features and text semantic features, the key information mis-cropping in long content is reduced, the integrity of sub-graph information is ensured; on the other hand, through the double positioning of boundary detection algorithm and boundary feature and the adjacent region feature verification, the boundary misjudgment problem in seamless splicing and complex background is solved, the coherence of text information is ensured, the semantic break problem is reduced, so as to improve the information utilization efficiency and optimize the user experience, and the stronger technical practicability is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a flowchart of an image processing method according to an example embodiment;

[0048] Figure 2 is a flowchart of an image processing method according to an example embodiment;

[0049] Figure 3 is a structural block diagram of an image processing device according to an example embodiment;

[0050] Figure 4 is an internal structure diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0051] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0052] The terms "first", "second", "third" in the embodiments of the present application are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second", "third" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0053] Reference to“an embodiment” herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase“in an embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all directed to the same embodiment, or to a single alternative embodiment. It is explicitly contemplated that embodiments described herein can be combined with each other in their individual and / or alternative embodiments.

[0054] In some embodiments, the image processing method provided in the embodiments of the present application can be applied in an electronic device. The electronic device can be any mobile terminal or fixed terminal. The terminal can refer to a device that provides voice and / or data connectivity to a user. Exemplarily, the terminal can be an Internet of Things terminal, such as a sensor device, a mobile phone or called“cellular” phone and a computer with an Internet of Things terminal, for example, can be a fixed, portable, pocket, handheld, computer built-in device.

[0055] In some embodiments, as shown in Figure 1 An image processing method is provided, and the method comprises the following steps:

[0056] S101, obtaining an original image to be processed; wherein the original image is spliced based on at least two sub-images.

[0057] In an embodiment, the original image can be an image containing text and graphics.

[0058] Exemplarily, the original image can include, but is not limited to, at least one of the following: an e-commerce commodity image, a social media chat image, a document scan image, and an advertising poster image.

[0059] S102, performing feature extraction on the original image to obtain a target feature; wherein the target feature includes at least one of the following: a visual feature, a text semantic feature, and a boundary feature.

[0060] In some embodiments, the target features can be extracted by artificially designed operators and mathematical models. For example, SIFT (Scale-Invariant Feature Transform) is used to detect key points (such as corner points, edge intersection points) in an image and extract local feature descriptors that are invariant to rotation, scale and illumination in different scale spaces. Alternatively, SURF (Speeded-Up Robust Features) is used to extract target features, which is an improved version of SIFT using integral images and Hessian matrix approximation to improve computational efficiency. Alternatively, HOG (Histogram of Oriented Gradients) is used to calculate the distribution of gradient directions in local regions of an image to capture shape and contour information of an object. Alternatively, a convolutional neural network (CNN) is used to extract target features. CNN automatically extracts hierarchical features from raw images, from low-level (such as edges, corner points), middle-level (such as textures, components) to high-level (such as object categories, semantic concepts) through multiple layers of learnable convolution kernels. In addition, attention mechanisms (such as SE, CBAM modules) and Transformer architectures can be introduced to enable the model to focus more on important regions in the image and improve feature expression capabilities.

[0061] In some embodiments, the feature extraction on the original image to obtain target features includes at least one of the following:

[0062] Firstly, local texture features in the visual features are extracted from the original image using a CNN branch in a CNN-Transformer hybrid model, and context-related features in the visual features are extracted using a Transformer branch in the CNN-Transformer hybrid model.

[0063] Secondly, candidate text features are extracted by recognizing the original image using optical character recognition, and the candidate text features are input into a natural language processing model to obtain the text semantic features.

[0064] Thirdly, the boundary features are extracted by analyzing the original image using frequency domain analysis and / or Hough transform.

[0065] In the embodiments of the present application, CNN-Transformer is a hybrid deep learning model combining convolutional neural network (CNN) and Transformer architecture.

[0066] In some embodiments, a CNN-Transformer hybrid model is adopted to construct a feature extraction framework. The CNN branch, with its strong ability to capture local features, performs layer-by-layer convolution operations on the original image, focusing on extracting local texture features within subgraphs (such as pattern textures in product detail images, bubble edge textures in chat records, etc.), and accurately depicting local visual details of the image. The shallow convolutional layers (such as the first 2 layers) perform edge detection on the pre-processed image through convolution operations, generating feature maps containing basic texture edges. The deep convolutional layers (layers 3-5) can use a 5x5 large convolution kernel, combined with a 2x2 max pooling layer (stride 2), which reduces the feature map size and computational complexity while expanding the receptive field (the receptive field range expands from the initial 3x3 to 21x21), achieving semantic aggregation of local textures (such as merging "fabric texture + pattern" in product images into complete product surface texture features, and merging "text font + bubble background color" in chat records into conversation unit texture features). Finally, through a global average pooling layer, the feature maps output by the deep convolutional layers are converted into fixed-dimensional local texture feature vectors (such as 512 dimensions), each vector corresponding to the local texture semantics of a sub-image in the long graph.

[0067] In some embodiments, the Transformer branch captures the contextual correlation features between different regions from the perspective of the entire long image through a self-attention mechanism (such as the visual logical correlation between the “main product image - parameter description image - user evaluation image” in the e-commerce long image, and the layout correlation between the “title - main text - annotation” in the document long image). Specifically, the local texture feature vectors output by the CNN branch are sequentially processed, and the feature vectors of all sub-images of the long image are arranged in the spatial position (from left to right, from top to bottom) as a feature sequence (such as N sub-images are divided from the long image, and an N x 512 feature sequence is generated); a position code is added to each feature vector to ensure that the Transformer can perceive the spatial order of the sub-blocks and reduce the correlation misjudgment caused by the disordered sequence; a multi-head self-attention layer (such as 6-8 heads) is constructed, and the feature sequence is projected into multiple different attention spaces; for each attention head, the attention weights between all feature vectors are calculated (such as the weight between the main product image feature vector and the parameter description image feature vector is high, and the weight between the main product image feature vector and the chat record feature vector is low), and a feature vector containing global correlation information is generated through weighted summation; residual connection and layer normalization are introduced to alleviate the gradient attenuation problem in the training process and ensure that the model can stably learn the correlation relationship between the long-distance sub-blocks in the long image. A feedforward neural network (composed of two linear layers and a GELU activation function) is connected after the multi-head self-attention layer, which performs nonlinear transformation and dimension adjustment on the global correlation features, strengthens the key correlation information, and finally outputs a fixed-dimension contextual correlation feature vector (consistent with the dimension of the output vector of the CNN branch, such as 512 dimensions).

[0068] In one embodiment, the computer device ensures that the two features of each sub-image can be accurately matched by one-to-one matching the local texture feature vectors of the CNN branch and the contextual correlation feature vectors of the Transformer branch according to the corresponding sub-image positions. An adaptive weight fusion strategy can also be used to fuse the two features to output the fused visual features.

[0069] In the embodiments of the present application, Optical Character Recognition (OCR) is a technology that automatically converts the text content (such as the text in scanned documents, photos, and PDF files) in an image into editable and searchable text data.

[0070] In the embodiments of the present application, Natural Language Processing (NLP) is a computing model that enables computers to understand, analyze, process, and generate human natural language.

[0071] In one embodiment, the original image is first subjected to full-text scanning by optical character recognition (OCR) technology, all text information in the image is accurately extracted, and candidate text features (covering various texts such as commodity prices, specifications, chat contents, document paragraphs, etc.) are formed; then the candidate text features are input into a pre-trained natural language processing (NLP) model (such as BERT, RoBERTa, etc.), the text is segmented, semantic coding and context association analysis are performed by the model, the semantic logic behind the text is mined, and text semantic features are obtained.

[0072] In one embodiment, for feature extraction of the subgraph splicing boundary in the long graph, frequency domain analysis or Hough transform technology, or a combination of the two, can be used to achieve complementary advantages. The frequency domain analysis converts the original image from the spatial domain to the frequency domain through Fourier transform, uses the energy distribution difference of different regions in the frequency domain to identify the subtle pixel mutations (such as hidden edges in seamless splicing, texture transition boundaries in complex backgrounds) that may exist at the subgraph splicing position, i.e., spectral mutation features; the Hough transform converts the edge lines in the image to the parameter space, and accurately detects the horizontal or vertical splicing boundary that may exist in the long graph, i.e., linear boundary.

[0073] In one embodiment, the electronic device can first extract the boundary candidate region of the subtle pixel mutation by frequency domain analysis, narrowing the boundary search range; perform Hough transform in the boundary candidate region to accurately detect the linear boundary therein; if the boundary candidate region of the frequency domain analysis coincides with the linear boundary position of the Hough transform, such as an error of less than 5 pixels, it can be confirmed that the boundary is the real boundary of the subgraph image splicing, and the boundary feature is obtained.

[0074] S103, performing boundary detection on the original image by a boundary detection algorithm, and determining the boundary position of each subgraph image in combination with the boundary feature.

[0075] In one embodiment, the electronic device implements multi-scale edge enhancement by adopting a U-Net++ architecture, divides the original long image into blocks with a size of 512x512 pixels (an overlap rate of 15% can be used to avoid loss of boundary information between blocks), converts the RGB channels of each sub-block into grayscale and boundary feature fusion channels (for example, an edge enhancement map obtained by frequency domain analysis and a candidate straight line map obtained by Hough transform are used as additional input channels), forms a 4-channel input, and enables the model to perceive potential boundary clues during the initialization stage. The input sub-block is normalized (pixel values are scaled to [0, 1]) to reduce the interference of brightness differences on boundary detection, and data augmentation (such as random rotation by 15°, horizontal flipping, and brightness jitter) is used to expand the training samples and improve the robustness of the model to different scene long images. Furthermore, through the encoder, decoder, and dense skip connection, the shallow details and deep semantics are fused to realize pixel-level boundary prediction. The encoder (downsampling stage) can use 4-layer convolution blocks (ConvBlock), each containing 2 3x3 convolutions (ReLU activation) and 1 2x2 max pooling (the step can be 2), and the number of convolution kernels gradually increases from 64 to 512 to expand the receptive field and focus on capturing deep semantic features. The decoder (upsampling stage) can correspondingly set 4 deconvolution blocks (UpConvBlock), each of which expands the feature map size by 2 times through 2x2 deconvolution, while reducing the number of convolution kernels from 512 to 64 to focus on restoring shallow detail features. Finally, intermediate convolution blocks are added between the encoder and the decoder to form a dense connection network. For example, the shallow features output by the first layer of the encoder are not only directly connected to the fourth layer of the decoder, but also processed by two intermediate convolution blocks and then connected to the third layer and the second layer of the decoder, respectively, to realize multi-scale transmission and semantic enhancement of shallow features and reduce the problem of shallow details being covered by deep semantics in traditional connection.

[0076] In some embodiments, the electronic device performs boundary detection on the original image through the boundary detection algorithm, obtains a boundary probability map, calculates the overlap rate (the number of overlapping pixels / the total number of boundary pixels) of each boundary detected by the U-Net++ and the high-intensity area of the edge enhancement map obtained by frequency domain analysis, and if the overlap rate is greater than or equal to a target threshold (such as 80%), it indicates that the boundary corresponds to a real pixel mutation, which is a spectral mutation feature. At the same time, it is judged whether the boundary coincides with the candidate straight line detected by the Hough transform (such as a distance error of <3 pixels): if the boundary is a horizontal / vertical boundary and coincides with the candidate straight line, it is further confirmed as a linear boundary for sub-image splicing; if it is an irregular boundary (such as a curved splicing of a promotional poster), its effectiveness is verified through frequency domain overlap rate to determine the boundary position of each sub-image.

[0077] S104, verifying the boundary position based on the visual features and the text semantic features of the two adjacent regions of the boundary position, to obtain a verification result.

[0078] In some embodiments, the visual features include color features; and verifying the boundary position based on the visual features and the text semantic features of the two adjacent regions of the boundary position, to obtain a verification result, includes:

[0079] verifying the color features of the two adjacent regions of the boundary position by using a first branch network in a double-branch verification network, and determining that a first alternative verification result is qualified in a case where the color features of the two adjacent regions do not match;

[0080] verifying the text semantic features of the two adjacent regions of the boundary position by using a second branch network in the double-branch verification network, and determining that a second alternative verification result is qualified in a case where semantic information represented by the text semantic features of the two adjacent regions is inconsistent;

[0081] In a case where both the first alternative verification result and the second alternative verification result are qualified, determining that the verification result is qualified.

[0082] In one embodiment, first, the two adjacent regions of the boundary position are cut from the original image as input data: taking the boundary line as the center, expanding a fixed width (such as 20-50 pixels, adjusted according to the resolution of a long image) to the left and right sides to form a left verification region and a right verification region; the normalized color feature vectors of the left verification region and the right verification region are input into a first branch network (such as MobileNetV2); the first branch network first unifies the feature dimensions through convolution, then extracts color difference features through deep separable convolution, and finally uses a fully connected layer and a Softmax activation function in the last layer to output the probabilities of color feature matching and color feature mismatching; if the probability of color feature mismatching is greater than a first probability threshold (such as 0.5, 0.6, etc.), it can be determined that the first alternative verification result is qualified; and the color variance of the boundary region and the inside of the sub-image can be further calculated, and in a case where the variance is less than a variance threshold, the first alternative verification result is determined to be qualified.

[0083] In one embodiment, the text semantic features of the left verification area and the right verification area are subjected to difference processing to obtain a semantic difference vector; the semantic difference vector is input into a second branch network, the semantic difference of each dimension is analyzed through a multi-head self-attention mechanism, and the probability of semantic information consistency and semantic information inconsistency is output through a full connection layer and a Softmax activation function; if the probability of semantic information inconsistency is greater than a second probability threshold (such as 0.55, 0.7, etc.), it can be determined that the second alternative verification result is qualified, indicating that there is no strong correlation between the semantic information on both sides of the boundary; if there is no text on both sides, the second alternative verification result can be defaulted as qualified, that is, in the case of no text interference, the first alternative verification result is relied on.

[0084] In one embodiment, only when the first alternative verification result and the second alternative verification result are both qualified, the final boundary verification result is determined as qualified, confirming that the boundary is a real subgraph splicing boundary and can be used for subsequent segmentation; if any alternative result is unqualified, the final verification result is unqualified, and the boundary needs to be rechecked.

[0085] In the embodiments of the present application, through the double-branch verification process, the boundary difference in the visual level can be captured through color features, and the boundary difference in the content logic level can be captured through text semantic features, and the two are cooperated to realize accurate verification of the long graph boundary position, greatly reducing the risk of missegmentation, and providing a core guarantee for accurate segmentation of subsequent subgraphs.

[0086] S105, in the case where the verification result is qualified, segmenting the original image based on the boundary position to obtain the subgraph image.

[0087] In the above image processing method, on the one hand, through the closed loop process of multi-feature extraction, accurate boundary positioning, two-way feature verification, and segmentation in the case of verification, the limitations of traditional fixed size or simple rule cropping can be broken through, the segmentation boundary can be judged from the content logic by virtue of the fusion of visual features and text semantic features, the key information mis-cropping in the long graph content can be reduced, and the integrity of the subgraph information can be guaranteed; on the other hand, through the double positioning of the boundary detection algorithm and the boundary feature and the adjacent area feature verification, the boundary misjudgment problem in the seamless splicing and complex background can be solved, the coherence of the text information can be guaranteed, and the problem of semantic breakage can be reduced, so that the information utilization efficiency is improved and the user experience is optimized, and stronger technical practicability is achieved.

[0088] In some embodiments, the method further comprises:

[0089] By performing integrity analysis on the boundary position of each subgraph image, an alternative subgraph image with mis-cropping is determined; wherein the integrity analysis comprises at least one of semantic integrity analysis and contour integrity analysis;

[0090] The repair content in the candidate sub-image that matches the text semantic features of the adjacent area of the boundary position is generated through a diffusion model, and the details of the repair content are optimized through multiple rounds of iteration;

[0091] The optimized repair content is fused with the corresponding candidate sub-image to obtain a first target sub-image.

[0092] In some embodiments, the electronic device performs semantic coherence analysis on the text semantic features of each sub-image, calculates the text semantic correlation degree of adjacent sub-images through an NLP model, and determines semantic discontinuity in the case where the correlation degree is greater than a correlation threshold or the text is incomplete, and marks it as semantically incomplete. For example, the first sub-image text "good, I tomorrow" and the second sub-image text "shipment" have a correlation degree greater than a correlation threshold of 0.7, and the text is a half sentence / isolated keyword, so it can be determined that the semantics are incomplete.

[0093] In one embodiment, the Canny edge detection is used to extract the contour of the instance in the sub-image to generate a contour mask; the contour closure degree is judged by the ratio of the number of contour breakpoints to the total length of the contour, and in the case where the closure degree is less than a closure degree threshold, it is marked as incomplete contour; and in combination with the linear boundary feature, if the instance contour and the linear boundary feature of the sub-image exist non-natural coincidence, the incomplete contour can be further determined.

[0094] In one embodiment, if the sub-image satisfies the semantic incompleteness and / or the contour incompleteness, it can be determined as an optional sub-image with mis-cut; the key instance (such as the main body of the commodity, the document table, etc.) in the optional sub-image can be segmented at the pixel level through an image segmentation model (Segment Anything Model, SAM), the instance mask output by the SAM is compared with the boundary of the optional sub-image, and if the instance mask exists beyond the boundary of the optional sub-image, the exceeding area is the mis-cut area, and its coordinate range in the optional sub-image is recorded.

[0095] In one embodiment, a precise prompt word is generated based on the text semantic features and context features of the mis-cut region, an edge control model (Canny) and a color control model (Color) are loaded through a ControlNet (Stable Diffusion, SD model plug-in), the complete contour and color distribution adjacent to the mis-cut region are input, and it is ensured that the contour and color of the generated content are consistent with the existing region; the coordinate range of the mis-cut region is used as a generation region mask to limit the SD model to generate content only in the region, thereby avoiding damage to the original complete part of the sub-image; a sampler in the SD model generates initial repair content and optimizes details through multiple rounds of iteration to obtain optimized repair content; the repair content is accurately aligned according to the mis-cut region coordinates, and the edges of the repair content are feathered; natural transition of the repair content and the alternative sub-image is realized through Poisson blending to obtain a first target sub-image.

[0096] In one embodiment, the first target sub-image can be re-analyzed for integrity to determine whether the quality of the first target sub-image meets the integrity requirement.

[0097] In the embodiments of the present application, the repair of the alternative sub-image with mis-cut can improve the problem of incomplete content caused by mis-cut in traditional segmentation, and the combination of instance segmentation and diffusion model ensures that the repair content is highly consistent with the original scene in terms of semantics and vision, greatly improving the integrity and usability of the sub-images of long image segmentation.

[0098] In some embodiments, the method further comprises:

[0099] selecting a first image from the sub-images and determining sub-images other than the first image as content images; wherein the first image is an image used to provide a target migration image style;

[0100] extracting feature maps of the first image and each of the content images using a pre-trained convolutional neural network;

[0101] calculating statistical data based on each of the feature maps; the statistical data includes mean and standard deviation;

[0102] performing instance normalization processing on the feature maps of the content images, and adjusting the statistical data of the normalized feature maps based on the statistical data corresponding to the first image to obtain processed feature maps;

[0103] inputting the processed feature maps into an encoder to reconstruct a second target sub-image after style migration.

[0104] In some embodiments, a style reference image, i.e., the first image, is selected from the segmented sub-images, requiring its style features (color tone, texture, light and shadow, etc.) to be representative of the scene; if not specified, the most frequently occurring style image is selected as the first image by calculating the style feature similarity (such as color histogram distance, texture feature vector distance) of all sub-images.

[0105] In some embodiments, the first image and all content images are unified in size (such as 512x512 pixels) and converted to RGB format; the pixel values are standardized to adapt to the input requirements of the pre-trained CNN. The relu1_1, relu2_1, relu3_1, relu4_1 layers of VGG19 are selected to extract style features (focusing on style information such as texture and color tone), and the relu4_2 layer is selected to extract content features (focusing on content information such as object structure and outline). The first image is input into VGG19, and the feature maps of the above style layers are output; each content image is input into VGG19, and the content layer feature map and the style layer feature map are output. For the style layer feature map of the first image, the mean and standard deviation of each channel are calculated; for the content layer feature map of the content image, the mean and standard deviation of each channel are calculated; instance normalization is performed on the content layer feature map to eliminate the original style information; based on the mean and standard deviation of the first image, the normalized feature map is adjusted to inject the target style; a lightweight convolutional decoder (symmetric to the VGG19 encoder, containing 4 deconvolution layers and ReLU activation) is used, the input is the AdaIN processed feature map, and the reconstruction is obtained through deconvolution and upsampling operation. For the reconstructed candidate target sub-image, the style similarity (such as color tone mean difference, texture entropy difference) with the first image is calculated, if the difference exceeds the threshold (such as color difference > 10), the AdaIN operation is re-executed by adjusting the weight of the standard deviation, and finally the style transfer image consistent with the input image size, i.e., the second target sub-image, is output.

[0106] In the embodiments of the present application, by selecting a first image of a specific style as a style source, relying on a pre-trained convolutional neural network to accurately extract style and content features, and combining statistical data adjustment to achieve accurate style transfer and complete content preservation, on the one hand, the core information of the content image is ensured not to be destroyed by style transfer through instance normalization, and on the other hand, all content images are unified to the target style based on the statistical data of the first image. The second target sub-image obtained through the encoder reconstruction can form a content complete and style unified synergistic effect with the sub-image repaired by the method in the previous text, and can also improve the overall visual coordination and scene adaptability of the long image, avoiding user visual fatigue or information reception efficiency decline caused by style disorder.

[0107] In some embodiments, asFigure 2 As shown, the segmentation of the original image based on the boundary position comprises:

[0108] S1051, obtaining a platform identifier of a platform to be published and a content type of content to be published;

[0109] S1052, matching the platform identifier and the content type with a preset rule library to obtain a matching result; wherein the rule library comprises a plurality of platform / content type corresponding cutting candidate rules;

[0110] S1053, segmenting the original image according to the cutting candidate rule matched successfully in the matching result and the boundary position to obtain the sub-image.

[0111] In one embodiment, a structured rule library is first built for mainstream platforms to be published and common content types. Each platform identifier and content type combination forms a corresponding exclusive cutting candidate rule. The rule can include but is not limited to at least one of the following core requirements: size and proportion, content priority, output format, and special restrictions.

[0112] Optionally, the common content types can include but are not limited to at least one of the following: product details, chat records, user reviews, promotional posters, and document paragraphs.

[0113] For example, the size and proportion requirements of a product image can be 800x800 pixels and 1:1 ratio; the size and proportion requirements of a WeChat public number image can be 750 pixels wide and unlimited height for chat record images.

[0114] For example, the content priority of a product image can be to preferentially retain price / specification text; the content priority of a chat record sub-image can be to preferentially retain conversation bubbles and user avatars.

[0115] For example, a product image uses JPG compression format and a poster image uses PNG lossless format.

[0116] In one embodiment, a double-layer screening logic matching rule can be used. First, through platform identifier screening, the exclusive rule set of the target platform is located from the rule library; second, according to the content type, the corresponding cutting candidate rule of the current long image content type is found in the platform rule set. If multiple candidate rules correspond to the same content type, the final applicable cutting rule is further determined in combination with the content features of different sub-images in the long image.

[0117] In one embodiment, the electronic device can determine the target size / ratio range of each sub-image according to the matched cropping rule; then, based on the boundary position of the sub-image located by the boundary detection algorithm, determine whether the boundary meets the requirements of the rule; if the sub-image size corresponding to the boundary is within the rule range, the boundary can be used as the segmentation line; if the sub-image size corresponding to the boundary exceeds the rule limit, find the closest boundary position within the rule allowed size range for segmentation; at the same time, combined with the content priority in the cropping rule, if the segmentation line may cut off the key content, fine-tune the segmentation position to the nearest complete boundary to ensure that the sub-image meets the platform size and does not lose the core information.

[0118] In the embodiments of the present application, by associating the to-be-published platform identifier and the content type, the corresponding cropping candidate rule is matched from the rule library, so that the segmentation is no longer limited to fixed size, but accurately fits the differentiated specifications of different platforms and the display requirements of content types; at the same time, combined with the boundary position accurately positioned in advance, both the key content of the sub-image when segmentation is reduced and the output sub-image directly meets the publication standards of the target platform are achieved, without the need for additional manual format adjustment, which improves the publication efficiency and guarantees the user's viewing experience.

[0119] In some embodiments, the method further comprises:

[0120] Obtaining a feedback index of the sub-image after being published to the to-be-published platform; wherein the feedback index includes a click rate index, a satisfaction index, and a stay time index;

[0121] Inputting the feedback index and the original image into a cropping decision model to obtain a segmentation optimization result; wherein the cropping decision model is used to indicate the mapping relationship between the feedback index and the segmentation strategy;

[0122] Segmenting the original image based on the segmentation optimization result and the boundary position to obtain a segmented and optimized sub-image.

[0123] In some embodiments, a state space is first defined by fusing original image features (visual features and text semantic features) and historical feedback indicators (normalized click rate, dwell time, and satisfaction), and discrete actions such as boundary pixel fine-tuning, subgraph proportion switching, and content priority weight adjustment are designed. The effect and compliance are balanced by a reward function, a policy network (outputting action probability) with a full connection network and a value network (estimating state value) sharing features are built, trajectories of states, actions, rewards, and next states are collected through interaction sampling, and advantage functions are calculated based on the GAE method. The PPO clipping objective function is used to limit the amplitude of policy update, and the reward is converged after a predetermined number of epochs. The optimized segmentation strategy is then output by the trained clipping decision model, and fine segmentation is performed based on the boundary position detected in the early stage to generate optimized subgraphs. The model is iteratively trained by continuously collecting new feedback indicators to adapt the segmentation strategy to user preferences and platform characteristics, thereby improving core indicators such as click rate and dwell time in the long term.

[0124] In the embodiments of the present application, real feedback data such as click rate, satisfaction, and dwell time after subgraph publishing can be accurately captured to identify the shortcomings of segmented subgraphs in actual scenarios. Feedback indicators and segmentation strategies are mapped by the clipping decision model to correct the segmentation logic. Finally, the segmented subgraphs based on the optimization results can dynamically iterate the segmentation strategy based on user feedback. This not only improves the user appeal, user experience, and information transmission efficiency of subgraphs, but also allows the segmentation scheme to adapt to changes in platform algorithms and upgrades in user needs, forming a long-term and sustainable segmentation effect optimization mechanism.

[0125] In some embodiments, the electronic device can perform super-resolution reconstruction using an ESRGAN model to ensure that the cropped subgraphs remain high quality and have no obvious artifacts even when enlarged. And / or, the Zero-DCE algorithm can be used to enhance the color of the subgraphs, effectively solving the problem of overexposure or underexposure and improving the overall color level.

[0126] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0127] Based on the same inventive concept, the embodiments of the present application also provide an image processing device for implementing the image processing method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations of the image processing method described above, which will not be repeated here.

[0128] In one embodiment, as shown in Figure 3 An image processing device is provided, the device comprising:

[0129] An acquisition module 10 is configured to acquire an original image to be processed; wherein the original image is spliced based on at least two sub-images;

[0130] A feature extraction module 20 is configured to perform feature extraction on the original image to obtain target features; wherein the target features include at least one of visual features, text semantic features, and boundary features;

[0131] A boundary detection module 30 is configured to perform boundary detection on the original image by a boundary detection algorithm, and determine the boundary positions of the sub-images in combination with the boundary features;

[0132] A verification module 40 is configured to verify the boundary positions based on the visual features and the text semantic features of the adjacent regions on both sides of the boundary positions, and obtain a verification result;

[0133] A segmentation module 50 is configured to segment the original image based on the boundary positions to obtain the sub-images if the verification result is qualified.

[0134] In one embodiment, the device further comprises:

[0135] The first determining module is configured to determine the candidate sub-image with the mis-cropping by performing integrity analysis on the boundary position of each sub-image; wherein the integrity analysis comprises at least one of semantic integrity analysis and contour integrity analysis.

[0136] The repairing module is configured to generate repair content in the candidate sub-image matching the text semantic features of the adjacent area of the boundary position by a diffusion model, and optimize the detail features of the repair content by multiple rounds of iteration.

[0137] The fusion module is configured to fuse the optimized repair content and the corresponding candidate sub-image to obtain a first target sub-image.

[0138] In an embodiment, the apparatus further comprises:

[0139] The screening module is configured to screen a first image from the sub-images, and determine sub-images other than the first image as content images; wherein the first image is an image used to provide a target image style.

[0140] The feature map extraction module is configured to extract feature maps of the first image and each of the content images respectively by using a pre-trained convolutional neural network.

[0141] The calculation module is configured to calculate statistical data based on each of the feature maps; wherein the statistical data comprises mean and standard deviation.

[0142] The processing module is configured to perform instance normalization processing on the feature maps of the content images, and adjust the statistical data of the normalized feature maps based on the statistical data corresponding to the first image to obtain processed feature maps.

[0143] The reconstruction module is configured to input the processed feature maps into an encoder to reconstruct a second target sub-image after style migration.

[0144] In an embodiment, the segmentation module 50 is configured to perform the following steps:

[0145] Obtain a platform identifier of a platform to be published and a content type of content to be published;

[0146] Match the platform identifier and the content type with a preset rule library to obtain a matching result; wherein the rule library comprises a plurality of platform / content type pairs corresponding to cropping candidate rules;

[0147] Segment the original image according to the cropping candidate rules matched successfully in the matching result and the boundary position to obtain the sub-images.

[0148] In an embodiment, the apparatus further comprises:

[0149] The acquisition module 10 is configured to acquire a feedback index of the sub-image after the sub-image is published to a to-be-published platform, wherein the feedback index comprises a click rate index, a satisfaction index, and a stay time index.

[0150] The segmentation optimization module is configured to input the feedback index and the original image into a cropping decision model to obtain a segmentation optimization result, wherein the cropping decision model is used to indicate a mapping relationship between the feedback index and a segmentation strategy.

[0151] The segmentation module 50 is configured to segment the original image based on the segmentation optimization result and the boundary position to obtain a segmented sub-image.

[0152] In an embodiment, the feature extraction module 20 is configured to perform at least one of the following steps:

[0153] Firstly, a CNN branch in a CNN-Transformer hybrid model is used to extract local texture features in the visual features from the original image, and a Transformer branch in the CNN-Transformer hybrid model is used to extract context correlation features in the visual features.

[0154] Secondly, optical character recognition is used to extract candidate text features from the original image, and the candidate text features are input into a natural language processing model to obtain the text semantic features.

[0155] Thirdly, frequency domain analysis and / or Hough transformation are used to analyze and extract the boundary features from the original image.

[0156] In an embodiment, the visual features comprise color features, and the verification module 40 is configured to perform the following steps:

[0157] The first branch network in a double-branch verification network is used to verify the color features of the adjacent two side regions of the boundary position, and in a case where the color features corresponding to the adjacent two side regions do not match, a first alternative verification result is determined to be qualified.

[0158] The second branch network in the double-branch verification network is used to verify the text semantic features of the adjacent two side regions of the boundary position, and in a case where semantic information represented by the text semantic features corresponding to the adjacent two side regions is inconsistent, a second alternative verification result is determined to be qualified.

[0159] In a case where both the first alternative verification result and the second alternative verification result are qualified, the verification result is determined to be qualified.

[0160] Each of the modules in the image processing apparatus described above can be implemented wholly or partially by software, hardware, and a combination thereof. Each of the modules can be embedded in or independent of a processor of an electronic device in hardware form, or stored in a memory of the electronic device in software form so as to be invoked by the processor to perform operations corresponding to each of the modules.

[0161] In one embodiment, an electronic device, which can be a terminal, has an internal structure diagram as shown in Figure 4 The electronic device includes a processor, a memory, a communication interface, a display unit, and an input device connected by a method bus. The processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating method and a computer program. The internal memory provides an environment for running the operating method and the computer program in the non-volatile storage medium. The communication interface of the electronic device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved by WIFI, mobile cellular network, NFC (Near Field Communication), or other technologies. The computer program is executed by the processor to implement an image processing method. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad provided on the housing of the electronic device. The input device can also be an external keyboard, touchpad, or mouse, etc.

[0162] Those skilled in the art can understand that Figure 4 The structure shown in the above embodiment is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the diagram, or combine certain components, or have a different arrangement of components.

[0163] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in any of the above method embodiments.

[0164] In one embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps performed by the processor of the electronic device of any of the above embodiments.

[0165] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0166] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0167] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0168] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An image processing method, characterized by, The method comprises: acquiring an original image to be processed; wherein the original image is spliced based on at least two sub-images; performing feature extraction on the original image to obtain target features; wherein the target features include at least one of visual features, text semantic features and boundary features; performing boundary detection on the original image by a boundary detection algorithm, and determining the boundary positions of the sub-images in combination with the boundary features; verifying the boundary positions based on the visual features and the text semantic features of the adjacent regions on both sides of the boundary positions, to obtain a verification result; in the case that the verification result is qualified, segmenting the original image based on the boundary positions to obtain the sub-images.

2. The method of claim 1, wherein, The method further comprises: determining an alternative sub-image with mis-trim by performing integrity analysis on the boundary positions of each sub-image; wherein the integrity analysis includes at least one of semantic integrity analysis and contour integrity analysis; generating repair content in the alternative sub-image that matches the text semantic features of the adjacent regions of the boundary positions by a diffusion model, and optimizing the detail features of the repair content through multiple rounds of iteration; fusing the optimized repair content with the corresponding alternative sub-image to obtain a first target sub-image.

3. The method of claim 1, wherein, The method further comprises: selecting a first image from the sub-images, and determining sub-images other than the first image as content images; wherein the first image is an image used to provide a target image style for image migration; extracting feature maps of the first image and each content image by using a pre-trained convolutional neural network; calculating statistical data based on each feature map; the statistical data includes mean and standard deviation; performing instance normalization processing on the feature maps of the content images, and adjusting the statistical data of the normalized feature maps based on the statistical data corresponding to the first image, to obtain processed feature maps; inputting the processed feature maps into an encoder to reconstruct a second target sub-image after style migration.

4. The method of claim 1, wherein, The segmentation of the original image based on the boundary positions to obtain the sub-images comprises: acquiring a platform identifier of a platform to be published and a content type of content to be published; matching the platform identifier and the content type with a preset rule library to obtain a matching result; wherein the rule library includes a plurality of platform / content type corresponding trimming candidate rules; segmenting the original image based on the boundary positions and the trimming candidate rules matched successfully in the matching result to obtain the sub-images.

5. The method of claim 1, wherein, The method further comprises: acquiring feedback indicators after the sub-images are published to the platform to be published; wherein the feedback indicators include click rate indicators, satisfaction indicators and dwell time indicators; inputting the feedback indicators and the original image into a trimming decision model to obtain a segmentation optimization result; wherein the trimming decision model is used to indicate the mapping relationship between the feedback indicators and the segmentation strategy; segmenting the original image based on the segmentation optimization result and the boundary positions to obtain segmented and optimized sub-images.

6. The method of claim 1, wherein, The feature extraction on the original image obtains target features, and the target features include at least one of the following: The first kind: using a CNN branch in a CNN-Transformer hybrid model to extract local texture features in the visual features from the original image, and using a Transformer branch in the CNN-Transformer hybrid model to extract context correlation features in the visual features; The second kind: using optical character recognition to extract candidate text features from the original image, and inputting the candidate text features into a natural language processing model to obtain the text semantic features; The third kind: using frequency domain analysis and / or Hough transform to analyze and extract the original image to obtain the boundary features.

7. The method of claim 1, wherein, The visual features include color features; and the visual features and the text semantic features of the adjacent two side regions of the boundary position are used to verify the boundary position to obtain a verification result, including: A first network in a double-branch verification network is used to verify the color features of the adjacent two side regions of the boundary position, and in a case where the color features of the adjacent two side regions do not match, a first alternative verification result is determined to be qualified; A second network in the double-branch verification network is used to verify the text semantic features of the adjacent two side regions of the boundary position, and in a case where semantic information represented by the text semantic features of the adjacent two side regions is inconsistent, a second alternative verification result is determined to be qualified; In a case where the first alternative verification result and the second alternative verification result are both qualified, the verification result is determined to be qualified.

8. An image processing apparatus characterized by comprising: The device includes: An acquisition module configured to acquire an original image to be processed; wherein the original image is formed by splicing at least two sub-images; A feature extraction module configured to extract target features from the original image; wherein the target features include at least one of visual features, text semantic features, and boundary features; A boundary detection module configured to detect boundaries of the original image by using a boundary detection algorithm, and determine boundary positions of the sub-images based on the boundary features; A verification module configured to verify the boundary positions based on the visual features and the text semantic features of adjacent two side regions of the boundary positions to obtain a verification result; A segmentation module configured to segment the original image based on the boundary positions to obtain the sub-images in a case where the verification result is qualified.

9. An electronic device, comprising: The device includes a processor and a memory; wherein the memory is configured to store a computer program; and the processor is configured to implement the method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images