Image and e-commerce image processing method, device and storage medium

Abstract

The embodiments of the present application provide an image and e-commerce image processing method, device, and storage medium. The e-commerce image processing method provided by the embodiments of the present application, when performing saliency detection on e-commerce images, takes into account that, for e-commerce images, users are more likely to be attracted by text in e-commerce images than semantic objects; and that areas outside the text area still have a high degree of attention points. Saliency detection and text detection are performed on e-commerce images respectively; and based on the first saliency map obtained by saliency detection and the text probability map obtained by text detection, a second saliency map of the e-commerce image is generated, thereby realizing saliency detection for e-commerce image characteristics and helping to improve the accuracy of saliency detection of e-commerce images.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image and e-commerce image processing method, device and storage medium. Background Art

[0002] When it comes to online shopping, e-commerce imagery plays a crucial role in inspiring and guiding consumers. It plays a central role throughout the entire shopping process, introducing products to consumers, assisting with visual search, engaging consumers, and ultimately influencing their decisions. Due to the inherent nature of online shopping, the core goal of e-commerce imagery is to capture customers' attention. E-commerce imagery typically combines images and text to effectively attract and introduce customers. Therefore, predicting the saliency of e-commerce imagery is crucial for providing consumers with enhanced guidance and shopping experiences.

[0003] Existing saliency prediction works almost all focus on natural images. However, due to the fundamental differences in the design goals of e-commerce images and natural images, existing saliency prediction methods are insufficient in predicting the saliency of e-commerce images. Summary of the Invention

[0004] Various aspects of the present application provide an image and e-commerce image processing and model training method, device, and storage medium for achieving saliency prediction of e-commerce images.

[0005] This embodiment of the present application provides an e-commerce image processing method, including:

[0006] Get e-commerce images;

[0007] Extracting significant features of the e-commerce image to obtain significant features of the e-commerce image;

[0008] performing saliency detection on the e-commerce image according to the saliency feature to obtain a first saliency map of the e-commerce image;

[0009] performing text detection on the e-commerce image based on the significant features to obtain a text probability map of the e-commerce image;

[0010] A second saliency map of the e-commerce image is generated according to the first saliency map and the text probability map.

[0011] The present application also provides an image processing method, including:

[0012] Acquire an image to be processed; the image to be processed includes text information and images of other objects;

[0013] Extracting significant features of the image to be processed to obtain significant features of the image to be processed;

[0014] performing saliency detection on the image to be processed according to the saliency feature to obtain a first saliency map of the image to be processed;

[0015] Performing text detection on the image to be processed according to the salient features to obtain a text prediction probability map of the image to be processed;

[0016] A second saliency map of the image to be processed is generated according to the first saliency map and the text prediction probability map.

[0017] The present application also provides a model training method, including:

[0018] Get e-commerce image samples;

[0019] With the loss function minimized as the training objective, the initial model of the saliency detection model is trained using e-commerce image samples.

[0020] During the model training process, multiple cascaded network layers in the backbone network are used to extract significant features of e-commerce image samples to obtain predicted significant features corresponding to the multiple network layers;

[0021] Inputting the predicted saliency features corresponding to the multiple network layers into a text detection head to perform text detection to obtain a text prediction probability map;

[0022] Inputting the predicted saliency features corresponding to the last layer of the multiple network layers into a saliency detection head for saliency detection to obtain a predicted saliency map;

[0023] The loss function is determined based on the difference between the text prediction probability map and the text true value map, the difference between the predicted significance map and the true value significance map, and the difference between the predicted significance features and the true value significance features corresponding to the multiple network layers; the true value significance feature corresponding to each network layer is the same size as the predicted significance feature corresponding to the network layer.

[0024] An embodiment of the present application further provides a computing device, comprising: a memory and a processor; wherein the memory is used to store a computer program;

[0025] The processor is coupled to the memory and is used to execute the computer program to perform the steps in the above-mentioned e-commerce image processing method, and / or image processing method, and / or model training method.

[0026] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned e-commerce image processing method, and / or image processing method, and / or model training method.

[0027] The e-commerce image processing method provided in the embodiment of the present application takes into account that, for e-commerce images, users are more likely to be attracted by text in e-commerce images than by semantic objects, and that areas outside the text region still have a high degree of attention. Saliency detection and text detection are performed on the e-commerce images respectively. Based on the first saliency map obtained by the saliency detection and the text probability map obtained by the text detection, a second saliency map of the e-commerce image is generated, thereby realizing saliency detection targeting the characteristics of the e-commerce image and helping to improve the accuracy of saliency detection of the e-commerce image. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0029] Figure 1a This is a schematic diagram of an e-commerce image;

[0030] Figure 1b This is a density map of points of interest for an e-commerce image provided by an embodiment of the present application;

[0031] Figure 2a and Figure 2b The distribution of the proportion of focus points other than the text area in the e-commerce image provided by the embodiment of this application;

[0032] Figure 3a A schematic diagram of the calculation process of the viewing angle provided in an embodiment of the present application;

[0033] Figure 3b for Figure 3a The view angle measurement results of the view angle calculation process provided;

[0034] Figure 4 A flowchart of an e-commerce image processing method provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of the process of performing online saliency detection on e-commerce images using the saliency detection model provided in an embodiment of the present application;

[0036] Figure 6 Schematic diagram of the saliency detection model architecture and model training process provided in the embodiments of this application;

[0037] Figure 7 A flowchart of an image processing method provided in an embodiment of the present application;

[0038] Figure 8 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0040] Traditional saliency prediction methods aim to predict human attention at the pixel level and mainly rely on low-level features, including contrast, color, brightness, and texture. However, none of the traditional saliency prediction methods involve saliency prediction of e-commerce (abbreviated as e-commerce) images. In the embodiments of the present application, e-commerce images refer to images produced in e-commerce for displaying product information, usually including product pictures and text. For example, pictures produced on online shopping platforms (such as websites, apps, etc.) for displaying products and product price information, etc.

[0041] Due to the inherent nature of online shopping, the core goal of e-commerce images is to attract customers' attention, which mainly includes two aspects: (1) attracting consumers to pay attention to a certain product while browsing the website, and promoting a certain brand; (2) attracting consumers to pay attention to specific areas in the image, such as the product or the information they want to convey (such as price reduction promotion, etc.). Therefore, the content of e-commerce images is usually a combination of pictures and text to achieve the goal of effectively attracting and introducing customers. For example, Figure 1a As shown in the figure, e-commerce images include: semantic objects (XX thin and light notebook) and text information (price reduction promotion words "Limited time special offer, 80 off for every 300 spent; lowest price: xxxx yuan") of e-commerce images, etc. Therefore, the saliency prediction of e-commerce images is of great significance for providing consumers with enhanced guidance information and shopping experience. Existing saliency prediction work almost all focuses on natural images, including bottom-up or top-down methods. However, due to the essential difference in the design goals of e-commerce images and natural images, existing methods have shortcomings in predicting the saliency of e-commerce images. For example, when predicting the saliency of images in existing methods, the object area in natural images is one of the most important high-level clues, resulting in the prediction that the key feature area of ​​the product in the e-commerce image is as salient as the brand text area.

[0042] Therefore, to solve the text priority problem in e-commerce images, it is urgent to develop a new method for e-commerce image saliency prediction. At the same time, the lack of e-commerce image datasets has also hindered the research of saliency prediction models. To this end, the present embodiment provides a method for e-commerce image saliency prediction.

[0043] In response to the problem of the lack of e-commerce image datasets, the embodiment of the present application establishes an eye tracking dataset for e-commerce images, which contains a variety of e-commerce images collected from online shopping platforms, and provides focus maps and true saliency maps through eye tracking technology. In order to study human perceptual behavior of e-commerce images, the inventors of the present application established an e-commerce image dataset, which includes: a large number of e-commerce images, including collected image focus points and text boundaries. In short, the eye tracking dataset for e-commerce images established by the present application includes: hundreds of thousands of gaze points in eye tracking experiments of multiple subjects, and tens of thousands of text borders annotated by multiple volunteers. The inventors of the present application analyzed the eye tracking dataset of e-commerce images using eye tracking technology and found that e-commerce images have the following characteristics:

[0044] (1) Feature 1: For e-commerce images, users are more likely to be attracted by text in e-commerce images than by semantic objects. In the embodiments of the present application, semantic objects in e-commerce images mainly refer to product images in e-commerce images.

[0045] According to previous studies, images of general natural scenes are more likely to attract visual attention to semantic objects. The inventors of this application first analyzed the saliency ground truth map of e-commerce images and found that the text area of ​​e-commerce images attracted a lot of attention. In order to further evaluate this feature, the semantic object boxes and text boxes in the database were detected by YOLOv5 and CRAFT, and then the density of attention points falling in the text area and object area was calculated respectively (value per 1000 pixels). Figure 1b As shown, for e-commerce images, the density of attention points in text areas is much higher than that in object areas, while the density in object areas is only slightly higher than that in random areas. This is primarily because the core design goal of e-commerce images is to showcase products. Therefore, the product objects often occupy the vast majority of the image area, but only a small portion of the objects attracts visual attention. These results indicate that in e-commerce images, users' visual attention is more easily drawn to text.

[0046] (2) Feature 2: Although text in e-commerce images can greatly attract visual attention, there is still a lot of visual attention outside the text area.

[0047] According to Feature 1, it can be seen that text can greatly attract the user's visual attention. The inventors of this application further calculated the number of attention points outside the text area. Figure 2a and Figure 2b Shows the proportion of focus points outside the text area. Figure 2a The ratio of focus points outside the text area of ​​e-commerce images with focus points in the text area; Figure 2b The ratio of focus points outside the text area of ​​e-commerce images without focus points in the text area. Figure 2a and Figure 2b In , each point represents an e-commerce image in the e-commerce image dataset, the horizontal axis represents the area ratio of the text area on the e-commerce image, and the vertical axis represents the ratio of the focus points outside the text area on the e-commerce image to the total number of focus points in the e-commerce image. Figure 2a As shown in Figure 2, for the vast majority of images in the e-commerce image dataset, approximately 40% to 70% of the attention points fall outside the text region. This means that within the text region, visual attention is attracted by other regions with bottom-up or top-down saliency.

[0048] exist Figure 2b In the present invention, the inventors also calculated the relationship between the area ratio of the text area without any focus points in each e-commerce image and the ratio of the focus points outside the text area in the e-commerce image to the total number of focus points in the e-commerce image. Figure 2b The results show that the text areas in this section are not clustered, meaning there's no specific range of text areas that attracts a large amount of visual attention. It's also worth noting that similar trends appear across different categories of e-commerce images.

[0049] above Figure 2a and Figure 2b The results show that users’ attention points on e-commerce images are distributed and scattered. Therefore, saliency prediction of e-commerce images is complex and cannot be solved simply by text detection methods.

[0050] Feature 3: In e-commerce images, visual attention among different user subjects is consistent, especially the attention within the text area.

[0051] To investigate the consistent characteristics of e-commerce images, the inventors measured visual consistency in the e-commerce image dataset by calculating the linear correlation coefficient (LCC) between the saliency of a single subject and that of other subjects. Specifically, they calculated the correlation between the ground-truth saliency map of a single individual and that of other individuals. As shown in Table 1, to gain a detailed understanding of the differences in consistency across different regions when subjects viewed an image, the inventors calculated the LCC between each individual and others for different regions, including the entire image, text regions, and object regions. Table 1 also lists the LCC values ​​for two other eye-tracking datasets (LEDOV and Hollywood) for comparison. Furthermore, Table 1 demonstrates the visual consistency of positional deviations in the e-commerce image dataset by measuring the LCC between the saliency maps of two randomly selected e-commerce images. The results in Table 1 indicate that the visual consistency in the e-commerce image dataset is similar to that of other eye-tracking datasets. This further demonstrates that users tend to focus on text when viewing e-commerce images.

[0052] Table 1 Consistency of different datasets

[0053]

[0054] Feature 4: Gaze shifts in e-commerce images are typically much larger than those in the fovea, indicating a tendency for non-local content in e-commerce images to attract visual attention. The fovea is the area of ​​the retina with the highest visual acuity (color and resolution).

[0055] Figure 3a Schematic diagram of the calculation process of the viewing angle provided in this application. Figure 3a As shown in Figure 1, the inventors of this application evaluated the gaze shift in the e-commerce image dataset by calculating the angle θ between two consecutive gazes of each subject. In the eye tracking experiment of this application, the size of the screen and the distance between the test subject (user) and the screen are fixed, so the angle of view can be calculated by trigonometric functions. Figure 3a In the figure, point O represents the gaze position of the fovea; points A and B represent the gaze positions of two consecutive visual shifts. The angle θ between two consecutive gazes can be calculated by trigonometric function. Figure 3a The calculation process of the viewing angle is shown to perform viewing angle measurement and obtain Figure 3b The visual angle measurement results shown in the figure are as follows. According to the research on the human visual system, human visual attention is only focused on a small area with a visual angle of no more than 2 degrees. Figure 3bThe visual angle measurement results shown in the figure show that only 25.8% of the subjects' gaze shifts toward e-commerce images were located in the fovea. Therefore, users experience a significant shift in their gaze for e-commerce images. This significant shift in gaze indicates that in e-commerce images, human attention is more easily drawn to non-local content. This may be because e-commerce images are originally designed to display all semantic objects and text within the image, rather than just a portion of it.

[0056] Based on the above-mentioned study on the gaze characteristics of subjects on e-commerce images, the e-commerce image processing method provided in the embodiments of the present application, when performing saliency detection on e-commerce images, takes into account that, for e-commerce images, users are more likely to be attracted by the text in e-commerce images than by semantic objects; and that the area outside the text still has a large degree of attention point. Saliency detection and text detection are performed on the e-commerce images respectively; and based on the first saliency map obtained by the saliency detection and the text probability map obtained by the text detection, a second saliency map of the e-commerce image is generated, thereby realizing saliency detection for the characteristics of the e-commerce image and helping to improve the accuracy of saliency detection of e-commerce images.

[0057] The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0058] It should be noted that the same reference numerals denote the same objects in the following drawings and embodiments, and therefore, once an object is defined in one drawing or embodiment, it does not need to be further discussed in the subsequent drawings and embodiments.

[0059] Figure 4 This is a flow chart of the e-commerce image processing method provided in the embodiment of the present application. Figure 4 As shown in Figure 2, the e-commerce image processing method mainly includes:

[0060] 401. Get e-commerce image.

[0061] 402. Extract salient features of the e-commerce image to obtain salient features of the e-commerce image.

[0062] 403. Perform saliency detection on the e-commerce image based on the saliency features to obtain a first saliency map of the e-commerce image.

[0063] 404. Perform text detection on the e-commerce image based on the saliency features to obtain a text probability map of the e-commerce image.

[0064] 405. Generate a second saliency map of the e-commerce image based on the first saliency map and the text probability map.

[0065] In an embodiment of the present application, in order to perform saliency detection on an e-commerce image, an e-commerce image may be obtained in step 401. The e-commerce image is any e-commerce image to be processed, and may be a two-dimensional image or a three-dimensional image. Of course, the e-commerce image may be an independent frame image, or any video frame in a video. Optionally, for a device that performs e-commerce image processing, an access request to an online shopping platform may be obtained; and an identifier of the image to be accessed may be obtained from the access request; thereafter, an e-commerce image corresponding to the identifier of the image to be accessed may be obtained from the server of the online shopping platform as the e-commerce image in step 401. For a description of the characteristics of the e-commerce image, please refer to the above content and will not be repeated here.

[0066] Furthermore, in step 402, salient features of the e-commerce image can be extracted to obtain salient features of the e-commerce image. Image saliency is an important visual feature in an image, reflecting the degree to which the human eye emphasizes certain areas of the image. Salient features can be expressed as a saliency feature map (Feature Map), which is a multi-dimensional feature vector.

[0067] Based on the above-mentioned characteristic analysis of e-commerce images, it can be seen that for e-commerce images, users are more likely to be attracted by the text in e-commerce images than by semantic objects; and there is still a high degree of attention outside the text area. Therefore, in an embodiment of the present application, when performing saliency detection on an e-commerce image, in step 403, saliency detection can be performed on the e-commerce image based on the saliency features to obtain a saliency map of the e-commerce image; and in step 404, text detection can be performed on the e-commerce image based on the saliency features to obtain a text probability map of the e-commerce image. Each probability value in the text probability map refers to the probability that each pixel in the e-commerce image is text. Among them, text detection can also be called text detection.

[0068] In this embodiment of the present application, the saliency detection in step 403 and the text detection in step 404 can be two non-intersecting branch networks or two branch networks sharing a common backbone network. In this embodiment of the present application, to improve the accuracy of subsequent saliency detection, steps 403 and 404 can be two branch networks sharing a common backbone network. Accordingly, step 402 can be implemented by extracting saliency features from the e-commerce image using the backbone network of the saliency detection model to obtain saliency features for the e-commerce image.

[0069] In some embodiments, the backbone network can be implemented as a neural network model, wherein the neural network model can be a machine learning model of any structure, including but not limited to: CNN, DNN, RNN, FCN and Transformer model.

[0070] Furthermore, based on characteristics 2 and 4 of the aforementioned e-commerce images, we can see that the focus information in e-commerce images is non-local. The Transformer model's attention mechanism allows the model to focus on global perceptual cues when making predictions. Therefore, the Transformer model can be selected as the backbone network for extracting salient features from e-commerce images. However, the computational complexity of extracting salient features using the Transformer model is relatively high, and the computational complexity issues associated with visual tasks remain unresolved. The self-attention mechanism requires calculating an N^2 relationship matrix for all N input tokens. Considering that visual information is inherently two-dimensional (images) or even three-dimensional (videos), a slightly higher resolution will result in a significantly higher computational load for the Transformer model.

[0071] To reduce the computational complexity of saliency detection, some embodiments of this application employ the Swin-Transformer model as the backbone network. The Swin-Transformer model addresses the high computational complexity of the Transformer model by proposing a layered architecture and shifting window design. It has been proven to reduce computational complexity in various visual tasks. The Swin-Transformer model improves efficiency through multi-scale layers and learning self-attention maps within a shifting window.

[0072] Among them, the model architecture of the saliency detection model is as follows Figure 5 and Figure 6 As shown in Figure 1, the layered architecture of the backbone network mainly refers to multiple cascaded network layers. Multiple refers to two or more layers. Figure 6 In the figure, only the number of network layers is 4, but it does not constitute a limitation. Figure 5 and Figure 6 As shown, the network layer may include: a windowed multi-head self-attention (Window Multi-head Self-Attention, W-MSA) module and a shifted window multi-head self-attention (Shifted-Window Multi-head Self-Attention, SW-MSA) module.

[0073] Based on the aforementioned saliency detection model, one implementation of step 403 is to use the saliency detection head of the saliency detection model to perform saliency detection on the e-commerce image based on saliency features, thereby obtaining a saliency map for the e-commerce image. Accordingly, one implementation of step 404 is to use the text detection head of the saliency detection model to perform text detection on the e-commerce image based on saliency features, thereby obtaining a text probability map for the e-commerce image.

[0074] In the embodiments of the present application, the saliency detection head and the text detection head can use a light and effective model to perform saliency map and text probability prediction. Figure 5 As shown, the saliency detection head may include: multiple dense blocks, a multi-scale information extraction module, and a deconvolution block (DeConv). Multiple refers to two or more. For example, the dense blocks can be three or four. Figure 5 The number of dense blocks is only 3 for illustration, but it does not constitute a limitation. Optionally, the multi-scale information extraction module can use an atrous spatial pyramid pooling (ASPP) structure to extract multi-scale information. The multi-scale information extraction module can use atrous convolutions with different sampling rates. For example, Figure 5 The saliency detection head of

[15] uses four kinds of dilated convolutions with different sampling rates. Figure 5 The dilated convolution can be implemented as a dilated convolution. Furthermore, a deconvolution block can be used to recover multi-scale information and obtain a saliency map. The number of deconvolution blocks can be multiple. Multiple refers to two or more. Figure 5 Three deconvolution blocks are shown in the figure, but this is not a limitation.

[0075] In the embodiments of the present application, the specific structure of the text detection head is not limited. In some embodiments, the text detection head can utilize the basic structure of CRAFT and can realize character-level text detection based on the idea of ​​segmentation. Figure 5 As shown, the text detection head may include: an upsampling block and an upconvolution block. After the saliency features output by the backbone network are upsampled and upconvolved, they are passed through the convolution module to output the character area score map (SOC) and affinity score map (AFF) of the e-commerce image. Among them, each score in the character area score map represents the probability that the corresponding pixel of the e-commerce image is the center of the character; the affinity score represents the probability of the center of the adjacent character area. Based on the basic architecture of the basic structure of CRAFT, in this embodiment, the text detection head of the saliency detection model can be used to perform character detection on the e-commerce image according to the saliency features to obtain the character area score map of the e-commerce image; and the connection relationship between the characters in the e-commerce image is detected according to the saliency features to obtain the affinity score map of the e-commerce image; further, the character area score map and the affinity score map can be determined as the text probability map of the e-commerce image.

[0076] Based on the above analysis of the characteristics of e-commerce images, it can be seen that for e-commerce images, users are more likely to be attracted to text in e-commerce images than to semantic objects; and there is still a high degree of attention outside the text area. Therefore, in step 405, a saliency map for the e-commerce image can be generated based on the saliency map and text probability map obtained in step 403. In this embodiment of the application, for ease of description and distinction, the saliency map obtained in step 403 is defined as the first saliency map, and the saliency map obtained in step 405 is defined as the second saliency map.

[0077] Specifically, the pixel values ​​of the same pixel coordinates in the text probability map and the first saliency map may be added to obtain the second saliency map of the e-commerce image.

[0078] The e-commerce image processing method provided in the embodiment of the present application takes into account that, for e-commerce images, users are more likely to be attracted by text in e-commerce images than by semantic objects, and that areas outside the text still have a high degree of attention. Saliency detection and text detection are performed on the e-commerce image respectively, and a second saliency map of the e-commerce image is generated based on the first saliency map obtained by the saliency detection and the text probability map obtained by the text detection, thereby realizing saliency detection targeting the characteristics of the e-commerce image and helping to improve the accuracy of saliency detection of the e-commerce image.

[0079] The specific implementation of the above steps 402 to 404 can be implemented using a saliency detection model. Figure 5 and Figure 6 As shown, the saliency detection model may include a backbone network, a saliency detection head, and a text detection head. The backbone network is used to extract saliency features from e-commerce images; the saliency detection head and the text detection head perform saliency detection and text detection on e-commerce images based on the saliency features, respectively. Before using the saliency detection model online, it must be trained. The following example illustrates the training process of a saliency detection model consisting of a backbone network, a saliency detection head, and a text detection head.

[0080] In the embodiment of the present application, in order to improve the network prediction performance, saliency information can be added to each SWN transformation block to promote the performance of network prediction. Figure 6 The saliency prediction features output by each network layer in the saliency detection head can be input to each SWN transformation block ( Figure 6 In the embodiment of the present application, for the last basic layer in each stage of the network layer, the embodiment of the present application proposes an attention loss L using a saliency map. a, to supervise the salient features learned in the backbone network. For the l-th network layer, the loss L a Expressed as:

[0081]

[0082] Where, in formula (1), M represents the total number of channels in the lth network layer. M represents the mth channel in the lth layer. m=1,2,…,M. A l,c Represents the self-attention map of the mth channel of the lth layer output, S l Represents a saliency map of the same size as the self-attention map of layer l, i.e., a saliency prediction feature. The saliency prediction features corresponding to multiple network layers are obtained by resizing the ground-truth saliency maps of e-commerce image samples. The size of the saliency prediction feature corresponding to each network layer is equal to the self-attention map output by that network layer.

[0083] In the above formula (1), cor(·) can be expressed as:

[0084] cor(X)=soft max(vec(X)·(vec(X) T ) (2)

[0085] In formula (2), vec(X) is the value of h l ×w l The matrix X is vectorized into a vector of w×h×l, and softmax(·) represents the softmax operation. In the embodiment of the present application, X in formula (2) is A in formula (1) l,c and S l .

[0086] The main purpose of the attention loss proposed in this application is to guide the backbone network proposed in this application (such as the SSwin-Transformer backbone network) to learn the non-local clues of e-commerce images based on human perception, while maintaining the diversity brought by the multi-head self-attention map. On the one hand, the promoted backbone network will focus on the non-local areas that humans pay the most attention to; this plays a vital role in tasks such as saliency prediction and text detection of e-commerce images, because people pay more attention to text areas when viewing e-commerce images. On the other hand, it also imposes a global and unified priority on the network, which is very important in multi-task learning. At the same time, the output features of the backbone network are enhanced by the global clues of multi-task learning, which is beneficial to subsequent saliency detection and text detection.

[0087] In the embodiment of the present application, the loss function of the saliency detection head can be represented by the difference between the saliency map predicted by the saliency detection head (i.e., the saliency prediction map) and the saliency ground truth map. For example, the Kullback-Leibler (KL) divergence between the saliency map predicted by the saliency detection head (i.e., the saliency prediction map) and the saliency ground truth map can be calculated. Accordingly, the loss function L of the saliency detection head is s It can be expressed as:

[0088] L s =KL(S p ||S gt ) (3)

[0089] In the above formula (3), S p S represents the saliency prediction map of the e-commerce image sample output by the saliency detection head; gt Represents the ground truth saliency map of e-commerce image samples.

[0090] Regarding text detection, in the embodiment of the present application, the text detection head can adopt the basic structure of CRAFT, based on the idea of ​​segmentation, and can achieve character-level text detection. Figure 6 As shown in the figure, the saliency prediction features of different resolutions from different network layers of the backbone network are upsampled, convolved, and connected to enhance the information aggregation between different resolutions. Then, the text probability prediction map is output through multiple convolution modules. Figure 6 In the text detection head, the character probability prediction map can be represented as a character region score map and an affinity score map. In the embodiment of the present application, the loss function of the text detection head can be represented by the difference between the character probability prediction map and the character probability ground truth map. For example, the loss between the character probability prediction map and the character probability ground truth map can be evaluated by the mean square error (MSE) between the character probability prediction map and the character probability ground truth map.

[0091] In the embodiment of the present application, since the text area score map and the affinity score map are usually sparse, directly adding their MSEs together may cause sample imbalance, resulting in zero outputs in almost all places. In order to overcome the sample imbalance problem, balanced MSE (BMSE) can be used to alleviate the training problem of network output being 0. Accordingly, when calculating the mean square error between the text probability prediction map and the text probability true value map, N can be randomly selected. pos positive sample images and N neg negative sample images. Among them, the positive sample images are e-commerce image samples with correct annotations based on the known saliency ground truth map and text probability ground truth map; the negative sample images are e-commerce image samples with incorrect annotations based on the known saliency ground truth map and text probability ground truth map. Accordingly, the calculation method of BMSE is expressed as:

[0092]

[0093] In formula (4), (P∪N) represents the set of positive and negative samples, and (i, j) represents the pixel with coordinates (i, j) in the e-commerce sample image. X(i, j) represents the predicted value of the text probability of pixel (i, j); Y(i, j) represents the true value of the text probability of pixel (i, j). N pos and N neg Represent the number of positive and negative samples respectively. Correspondingly, the loss function L of the text detection head t Expressed as:

[0094]

[0095] In formula (5) and They respectively represent the character region score prediction map and the character region score truth map of the e-commerce image sample output by the text detection head; and They respectively represent the character affinity score prediction graph of the e-commerce image samples output by the text detection head and the character affinity score truth graph of the e-commerce image samples.

[0096] Based on the above analysis, the loss function of the saliency detection model can be expressed as:

[0097] L=λ a L a +λ s L s +λ t L t (6)

[0098] In formula (6), λ a ,λ s and λ t Represents the loss weights of the backbone network, saliency detection head, and text detection head, which are used to adjust the proportion of losses in different modules. a , L t , L s Represent the losses of the backbone network, saliency detection head, and text detection head respectively.

[0099] Based on the above Figure 6 The schematic diagram of the architecture of the saliency detection module is shown in FIG. The model training process of the saliency detection model mainly includes the following steps:

[0100] S1. Obtain e-commerce image samples.

[0101] S2. Minimizing the loss function is used as the training objective, and e-commerce image samples are used to perform model training on the backbone network, the saliency detection head, and the text detection head;

[0102] S3. During the model training process, multiple cascaded network layers in the backbone network are used to extract significant features of e-commerce image samples to obtain significant prediction features corresponding to multiple network layers.

[0103] S4. Inputting the predicted saliency features corresponding to the multiple network layers into a text detection head for text detection to obtain a text probability prediction map of the e-commerce image sample.

[0104] S5. Inputting the saliency prediction features corresponding to the last layer of the multiple network layers into the saliency detection head for saliency detection to obtain a saliency prediction map.

[0105] The loss function is determined based on the difference between the predicted word probability map and the true word probability map, the difference between the predicted saliency map and the true saliency map, and the difference between the predicted saliency features and the true saliency features corresponding to multiple network layers. The true saliency features corresponding to each network layer are of the same magnitude as the predicted saliency features corresponding to that network layer. The specific representation of the loss function can be found in the relevant content of the above formulas (1) to (6), and will not be repeated here.

[0106] Since saliency prediction features of different resolutions generated by different network layers are added to the text detection head during the training process of the above-mentioned saliency prediction model, the prediction performance of the text detection head can be improved, which helps to improve the accuracy of subsequent text detection using the saliency prediction model, and further improves the accuracy of the saliency map of e-commerce images generated based on the text probability map obtained by text detection and the saliency map obtained by saliency detection.

[0107] In other embodiments, saliency prediction features corresponding to multiple network layers can also be input into the saliency detection head for saliency detection. As can be seen from Feature 1 of the e-commerce image above, text in e-commerce images attracts more attention than other areas. Therefore, text detection takes precedence over saliency detection. Based on this, saliency prediction features corresponding to the first K layers of multiple network layers can be input into the saliency detection head for saliency detection. Here, K is less than the number of network layers.

[0108] In the embodiment of the present application, according to the characteristics 1 and 2 of the above-mentioned e-commerce images, it can be seen that in e-commerce images, text detection and saliency detection are mutually reinforcing. Figure 6 As shown, in an embodiment of the present application, during the training phase of the saliency detection model, information output from the saliency header can be fed back to the input of the text detection header, and information output from the text detection header can be fed back to the input of the saliency header. This interactive information flow between the saliency detection header and the text detection header can improve saliency prediction and text detection learning. Based on this, the forward process of the above-mentioned saliency detection header becomes:

[0109]

[0110] SalHead(·) in formula (7) represents the saliency detection head, and Respectively represent the character area score prediction map output by the text detection head Character affinity score prediction graph The character region score prediction map and character affinity score prediction map are resized to be equal to the saliency prediction feature F output by the backbone network. ⊙ represents the element-wise product, i.e., the dot product.

[0111] Among them, f(·) in formula (7) can be expressed as:

[0112] f(x)=ρ(x-0.5)+1 (8)

[0113] Where x is 0≤x≤1 and ρ is the scaling factor. For equation (7), x in equation (8) is: In this way, compared with the background area with zero output, the detected text area with positive value can appropriately increase the importance of the feature at the corresponding position, so that the performance of the saliency detection head can be further improved through this additional information.

[0114] Similarly, the text probability prediction map output by the above text detection head can be expressed as:

[0115]

[0116] In formula (8), TextHead(·) represents the text detection head. It represents the saliency prediction map output by the saliency detection head, which is equal to the saliency prediction feature F output by the backbone network. For formula (9), x in formula (8) is:

[0117] Based on the information feedback mechanism between the above-mentioned text detection head and the saliency detection head, the training process of the above-mentioned saliency prediction model includes multiple training rounds. For any current training round, the saliency prediction features corresponding to the multiple network layers output by the backbone network in the current training round and the saliency prediction map output by the saliency detection head in the previous training round can be input into the text detection head for text detection to obtain the text probability prediction map output by the text detection head in the current training round; and the saliency prediction features corresponding to the last layer of the multiple network layers and the text probability prediction map output by the text detection head in the previous training round are input into the saliency detection head for saliency detection to obtain the saliency prediction map output by the saliency detection head in the current training round; wherein the current training round is any training round except the first training round.

[0118] Specifically, the first dot product between the saliency features corresponding to the multiple network layers output by the backbone network in the current training round and the saliency prediction map output by the saliency detection head in the previous training round can be calculated. The first dot product is then input into the text detection head for text detection, resulting in a text probability prediction map output by the text detection head in the current training round. Furthermore, the second dot product between the saliency prediction features corresponding to the last layer of the multiple network layers and the text probability prediction map output by the text detection head in the previous training round is calculated, and the second dot product is input into the saliency detection head for saliency detection, resulting in a saliency prediction map output by the saliency detection head in the current training round.

[0119] It should be noted that in the first training round, according to the size of the significant prediction feature output by the backbone network in the first training round, a full-1 matrix equal to the size of the significant prediction feature is determined as the result of the text header output before the first training round, and a rough prediction S is obtained. p , and then feed it back into the text detection head to obtain the text probability prediction map output by the text detection head in the first training round and Then, by detecting the text obtained and This is then fed back to the saliency detection head to obtain a more refined saliency prediction map. It should be noted that although iterating this process may result in further enhanced predictions, as the number of iterations increases, the computational complexity increases while the gain decreases. To simplify computational complexity, in an embodiment of the present application, information of a set number of times can also be fed back to the text detection head and the saliency detection head. The set number of times can be less than the set number of training rounds. For example, the set number of times can be 1, 2, or 3, etc.

[0120] Regarding the information feedback mechanism between the above-mentioned text detection head and the saliency detection head, since saliency prediction features of different resolutions generated by different network layers are added to the text detection head during the training process of the above-mentioned saliency prediction model, the prediction performance of the text detection head can be improved; on the other hand, since the output information of the text detection head is fed back to the saliency detection head, the saliency prediction features of different resolutions generated by different network layers are also added to the saliency detection head, which also helps to improve the accuracy of subsequent saliency detection using the saliency prediction model.

[0121] The above embodiment exemplifies the training process of a saliency detection model. After the saliency detection model is trained, the saliency detection model can be used to perform saliency detection and text detection on e-commerce images to obtain a first saliency map and a text probability map for the e-commerce image. Furthermore, a second saliency map for the e-commerce image can be generated based on the first saliency map and text probability map output by the saliency detection model. Because the second saliency map incorporates the e-commerce image, users are more likely to focus on text in the e-commerce image than on semantic objects. Furthermore, areas outside the text region still have a high degree of attention. Therefore, the e-commerce image saliency detection method provided in this embodiment of the application is compatible with the characteristics of e-commerce images and can improve the accuracy of e-commerce image saliency detection compared to traditional saliency detection methods.

[0122] In an embodiment of the present application, for an e-commerce image, after obtaining the saliency map of the e-commerce image, the code rate of the e-commerce image can be allocated based on the second saliency map of the e-commerce image to obtain the code rate allocation result of the e-commerce image; thereafter, the e-commerce image can be encoded based on the code rate allocation result to achieve perceptual optimization of image encoding. Furthermore, the encoded e-commerce image can be transmitted to the image request end. Since the second saliency map of the e-commerce image reflects people's attention information on different areas of the e-commerce image, the code rate of the e-commerce image can be allocated based on the second saliency map of the e-commerce image to achieve encoding code rate allocation for areas of high attention and areas of low attention, so that areas of high attention are allocated a higher code rate, which can achieve perceptual optimization of image encoding.

[0123] In the embodiment of the present application, the specific implementation method of allocating bitrate to the e-commerce image based on the second saliency map of the e-commerce image is not limited. In some embodiments, the e-commerce image can be segmented to obtain multiple image blocks. Multiple refers to 2 or more. The image block can be called a coding tree unit (CTU). A CTU consists of a pixel block of size N*N. In some embodiments, N is 128 (for current 4k / 8k ultra-high-definition images). In the embodiment of the present application, the specific implementation method of image segmentation for the e-commerce image is not limited. In some embodiments, the method of image segmentation for the e-commerce image may include: one or more of a threshold-based segmentation method, an edge detection-based segmentation method, a region-based segmentation method, and a deep learning-based segmentation method, but is not limited thereto.

[0124] Furthermore, based on the second saliency map of the e-commerce image obtained in step 405, bitrate allocation can be performed on the multiple image blocks to obtain target bitrates for the multiple image blocks, which serve as the bitrate allocation results for the e-commerce image. Furthermore, image encoding can be performed on the e-commerce image based on the target bitrates for the multiple image blocks.

[0125] In some embodiments, the goal is to minimize the coding distortion of e-commerce images. With the bitrate of multiple image blocks equal to a set target bitrate as a constraint, a rate-distortion optimization method is used to calculate the initial bitrate of multiple image blocks, minimizing the overall perceptual distortion of the e-commerce image. Accordingly, the calculation method can be expressed as:

[0126]

[0127] Among them, d i and r i where represents the distortion and initial bitrate of the i-th image block, respectively. M represents the total number of image blocks corresponding to the current e-commerce image. R represents the target number of bits for encoding the e-commerce image and can be flexibly set based on actual needs. "st" specifies that the total number of bits for the e-commerce image must be equal to R.

[0128] In some embodiments, the image distortion and the bit rate satisfy a certain curve relationship. For example, the image distortion and the bit rate satisfy a hyperbolic function relationship. The curve relationship between the image distortion and the bit rate can be expressed as:

[0129] D(r i )=C*r i -K (11)

[0130] In formula (11), C and K are constants. Based on the above formulas (10) and (11), the initial bit rate r of each image block can be obtained by jointly solving i .

[0131] Furthermore, the initial bit rates of the multiple image blocks may be adjusted according to the second saliency map of the e-commerce image to obtain target bit rates of the multiple image blocks.

[0132] Specifically, saliency subgraphs corresponding to multiple image blocks can be determined from the second saliency map; then, saliency weights for the multiple image blocks can be calculated based on the saliency subgraphs corresponding to the multiple image blocks. Optionally, the sum of the saliency values ​​in the saliency subgraph corresponding to each image block can be calculated; and then, from the multiple image blocks, a target image block with the largest sum of corresponding saliency values ​​can be selected. For any image block A, the ratio of the sum of the saliency values ​​corresponding to the image block A to the sum of the saliency values ​​corresponding to the target image block can be calculated as the saliency weight for the image block A.

[0133] In some embodiments, to ensure minimal perceptual distortion of the image, the rate allocation of non-critical coding tree units can be reduced based on the visual perception model and the R-lambda rate control algorithm to reduce the perceptual distortion of the e-commerce image. Specifically, first, the saliency map of the current image is output based on the visual perception model. The saliency weight value of each CTU block of the current image is calculated in units of CTU blocks, which is used as the subjective weight w for perceptual coding. i When calculating the bitrate of each image block within the encoder, the initial bitrate of the image block is adjusted according to the significance weight of the corresponding image block to obtain the target bitrate of each image block. The overall bitrate is shown in formula (12):

[0134]

[0135] In formula (12), r i *(w i +w base ) represents the target bit rate of image block i. i is the saliency weight of image block i. base is the preset correction weight. Mainly due to the saliency weight w of image block i i The ratio of the sum of the saliency values ​​corresponding to the image block i to the sum of the saliency values ​​corresponding to the target image block. The sum of the saliency values ​​corresponding to the image block i is sometimes 0. In order to prevent the sum of the saliency values ​​corresponding to the image block i from being 0 and affecting the subsequent encoding effect, we set w base Perform weight correction on the sum of the saliency values ​​corresponding to the image block i. base It can be 0.5.

[0136] The size of the saliency weight map corresponding to the e-commerce image is the resolution of the e-commerce image divided by the size of the image block. Accordingly, the saliency weight map corresponding to the e-commerce image can be expressed as:

[0137] h w =(h+size-1) / size (13)

[0138] w w =(w+size-1) / size (14)

[0139] w in formulas (13) and (14) w and h w They represent the width and height of the saliency weight map respectively, and size is the size of the image block, such as 128 in video coding (VVC).

[0140] After determining the significance weight of each image block, the initial bit rates of the multiple image blocks can be adjusted using the significance weights of the multiple image blocks, with the sum of the bit rates of the multiple image blocks being equal to the set target bit number as a constraint, to obtain the target bit rates of the multiple image blocks. The specific adjustment method can be found in the above formula (12).

[0141] Afterwards, the e-commerce image can be encoded according to the target bit rate of the multiple image blocks. Specifically, the image coding parameters of the multiple image blocks can be calculated based on the target bit rate of the multiple image blocks and the curve relationship between the bit rate and the image coding parameter. Among them, the image coding parameter can be expressed as: quantization parameter (QP). Accordingly, the target bit rate of each image block and the curve relationship between the bit rate and the image coding parameter can be implemented as the following equations (15) and (16). The quantization parameter QP of the corresponding CTU block is calculated by the target bit rate for encoding. As shown in equations (3.10) and (3.11):

[0142]

[0143] QP i =4.2005·lnλ i +13.7122 (16)

[0144] In the above formula (15), λ i The coefficient of the curve between the distortion of image block i and the target bit rate of image block i. i and β i is a coefficient related to the content of image block i. bpp i represents the pixel depth of image block i, which can be calculated based on the target bit rate of image block i. The calculation formula is as follows:

[0145]

[0146] In formula (17), R i represents the target bitrate of image block i. f represents the frame rate. w i and h i denote the width and height of image block i respectively.

[0147] Based on the above equations (15)-(17), the image coding parameters corresponding to each image block, such as the quantization parameter QP, can be obtained. Furthermore, the image coding parameters of multiple image blocks can be used to perform image coding on the multiple image blocks to perform image coding on the e-commerce image.

[0148] In the encoding process of e-commerce images in the embodiments of the present application, based on the aforementioned information about people's attention to different areas of the image, the bit rate of the areas of the e-commerce image is redistributed, so that areas of high attention are allocated a higher bit rate, and areas of low attention are allocated a lower bit rate. While ensuring the encoding quality of areas of high attention, the encoding bit rate of areas of low attention can be reduced. Compared with traditional encoders, the encoding bit rate of e-commerce images can be reduced. The inventors of this application conducted perceptual coding verification on Versatile Video Coding (VVC), and the bit rate of images after perceptual coding can be saved by more than 20% compared to standard encoders.

[0149] In addition to being applicable to saliency detection and encoding of e-commerce images, the image processing methods provided in the embodiments of this application can also be used to detect and encode saliency of other images containing text and semantic objects. For example, video images with subtitles, human-computer interaction interfaces (such as UI interfaces), electronic posters, etc. The following is an illustrative description of the application of the image processing methods provided in the embodiments of this application to other application scenarios.

[0150] Figure 7 This is a flow chart of the image processing method provided in the embodiment of the present application. Figure 7 As shown, the method mainly includes:

[0151] 701. Acquire an image to be processed. The image to be processed includes text information and images of other objects.

[0152] 702. Perform salient feature extraction on the image to be processed to obtain salient features of the image to be processed.

[0153] 703. Perform saliency detection on the image to be processed based on the saliency features to obtain a first saliency map of the image to be processed.

[0154] 704. Perform text detection on the image to be processed based on the saliency features to obtain a text probability map of the image to be processed.

[0155] 705. Generate a second saliency map of the image to be processed based on the first saliency map and the text probability map.

[0156] In this embodiment, the image to be processed can be any image including text information and other objects, for example, a video frame image with subtitles, a UI interface, a pop-up window, an electronic poster, etc.

[0157] In this embodiment, the image to be processed has similar characteristics to the e-commerce image. Based on the fact that users are more likely to be drawn to text in the image to be processed than to semantic objects, and that there is still a high degree of attention outside the text area, this embodiment performs saliency detection and text detection on the image to be processed separately. Based on the first saliency map obtained from the saliency detection and the text probability map obtained from the text detection, a second saliency map for the image to be processed is generated, thereby implementing saliency detection tailored to the characteristics of the image to be processed and helping to improve the accuracy of saliency detection for the image to be processed. The specific implementation methods for performing saliency detection and text detection on the image to be processed can be found in the aforementioned section on saliency detection and text detection for e-commerce images, and will not be elaborated upon here.

[0158] It should be noted that the execution entity of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution entity of steps 401 and 402 can be device A; for another example, the execution entity of step 401 can be device A, and the execution entity of step 402 can be device B; and so on.

[0159] In addition, some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear in the present invention or may be executed in parallel. The sequence numbers of the operations, such as 403 and 404, are merely used to distinguish between different operations and do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0160] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned e-commerce image processing method, and / or image processing method, and / or model training method.

[0161] Figure 8 This is a schematic diagram of the structure of the computing device provided in the embodiment of the present application. In the embodiment of the present application, the implementation form of the computing device is not limited. Optionally, the computing device can be implemented as a single server, a cloud server array, etc.; of course, the computing device can also be implemented as a terminal device such as a mobile phone or a computer. Figure 8 As shown, the computing device includes: a memory 80a and a processor 80b; wherein the memory 80a is used to store computer programs;

[0162] The processor 80b is coupled to the memory 80a and is configured to execute a computer program for: acquiring an e-commerce image; performing saliency feature extraction on the e-commerce image to obtain saliency features of the e-commerce image; performing saliency detection on the e-commerce image based on the saliency features to obtain a first saliency map of the e-commerce image; performing text detection on the e-commerce image based on the saliency features to obtain a text probability map of the e-commerce image; and generating a second saliency map of the e-commerce image based on the first saliency map and the text probability map.

[0163] In some embodiments, when the processor 80b extracts salient features from the e-commerce image, it is specifically configured to: extract salient features from the e-commerce image using the backbone network of the saliency detection model to obtain salient features of the e-commerce image.

[0164] Accordingly, when the processor 80b performs saliency detection on the e-commerce image to obtain the first saliency map of the e-commerce image, it is specifically used to: use the saliency detection head of the saliency detection model to perform saliency detection on the e-commerce image according to the saliency features to obtain the first saliency map of the e-commerce image.

[0165] Optionally, when performing text detection on an e-commerce image, the processor 80b is specifically configured to: perform text detection on the e-commerce image according to saliency features using a text detection head of a saliency detection model to obtain a text probability map of the e-commerce image.

[0166] Furthermore, when the processor 80b uses the text detection head of the saliency detection model to perform text detection on the e-commerce image based on the saliency features, it is specifically used to: use the text detection head of the saliency detection model to perform character detection on the e-commerce image based on the saliency features to obtain a character area score map of the e-commerce image; each score in the character area score map represents the probability that the corresponding pixel of the e-commerce image is the center of the character; detect the connection relationship between characters in the e-commerce image based on the saliency features to obtain a character affinity score map of the e-commerce image; the character affinity score represents the probability of the centers of adjacent character areas; and determine the character area score map and the character affinity score map as a text probability map.

[0167] In some embodiments, the processor 80b is further used to: obtain e-commerce image samples; use the e-commerce image samples to perform model training on the backbone network, saliency detection head and text detection head with minimization of the loss function as the training objective; during the model training process, use the multiple network layers cascaded in the backbone network to extract saliency features of the e-commerce image samples to obtain saliency prediction features corresponding to the multiple network layers; input the saliency prediction features corresponding to the multiple network layers into the text detection head for text detection to obtain a text probability prediction map of the e-commerce image samples; input the saliency prediction features corresponding to the last layer of the multiple network layers into the saliency detection head for saliency detection to obtain a saliency prediction map; wherein the loss function is determined based on the difference between the text probability prediction map and the text probability true value map, the difference between the saliency prediction map and the saliency true value map, and the difference between the saliency prediction features corresponding to the multiple network layers and the saliency true value features; the saliency true value feature corresponding to each network layer is the same size as the saliency prediction feature corresponding to the network layer.

[0168] Among them, the difference between the saliency prediction map and the saliency truth map is represented by the divergence between the saliency prediction map and the saliency truth map; the difference between the saliency prediction features corresponding to multiple network layers and the saliency truth features is represented by the divergence between the saliency prediction features corresponding to multiple network layers and the saliency truth features; the saliency truth features corresponding to each network layer are obtained by resizing the saliency truth map.

[0169] The above-mentioned e-commerce image samples include: positive sample images and negative sample images; the positive sample images are e-commerce image samples with correct annotations for known saliency true value maps and text probability true value maps; the negative sample images are e-commerce image samples with incorrect annotations for known saliency true value maps and text probability true value maps.

[0170] Optionally, the model training process includes multiple training rounds. The processor 80b is further configured to: for the current training round, input the saliency prediction features corresponding to the multiple network layers output by the backbone network in the current training round and the saliency prediction map output by the saliency detection head in the previous training round into the text detection head for text detection, thereby obtaining a text probability prediction map output by the text detection head in the current training round; input the saliency prediction features corresponding to the last layer of the multiple network layers and the text probability prediction map output by the text detection head in the previous training round into the saliency detection head for saliency detection, thereby obtaining a saliency prediction map output by the saliency detection head in the current training round; the current training round is any training round other than the first training round.

[0171] Optionally, when the processor 80b inputs the saliency features corresponding to multiple network layers output by the backbone network in the current training round and the saliency prediction map output by the saliency detection head in the previous training round into the text detection head for text detection, it is specifically used to: calculate the first dot product between the saliency features corresponding to multiple network layers output by the backbone network in the current training round and the saliency prediction map output by the saliency detection head in the previous training round, and input the first dot product into the text detection head for text detection.

[0172] Correspondingly, when the processor 80b inputs the saliency features corresponding to the last layer of multiple network layers and the text probability prediction map output by the text detection head in the previous training round into the saliency detection head for saliency detection, it is specifically used to: calculate the second dot product between the saliency prediction features corresponding to the last layer of multiple network layers and the text probability prediction map output by the text detection head in the previous training round, and input ... detection head and the text probability prediction map.

[0173] In some embodiments, when the processor 80b generates the second saliency map of the e-commerce image based on the first saliency map and the text probability map, it is specifically configured to add the pixel values ​​of the same pixel coordinates in the text probability map and the first saliency map to obtain the second saliency map of the e-commerce image.

[0174] In an embodiment of the present application, the processor 80b is further configured to: perform image segmentation on the e-commerce image to obtain a plurality of image blocks; perform bit rate allocation on the plurality of image blocks according to the second saliency map to obtain a target bit rate for the plurality of image blocks; and perform image encoding on the e-commerce image according to the target bit rates for the plurality of image blocks.

[0175] Optionally, processor 80b is further configured to calculate initial bit rates for the multiple image blocks using a rate-distortion optimization method, with the goal of minimizing e-commerce image coding distortion and with the constraint that the sum of the bit rates of the multiple image blocks equals a set target bit count. Accordingly, when using the second saliency map, processor 80b is specifically configured to adjust the initial bit rates for the multiple image blocks based on the second saliency map to obtain target bit rates for the multiple image blocks.

[0176] Furthermore, when adjusting the initial bit rates of the multiple image blocks based on the second saliency map, the processor 80b is specifically configured to: determine saliency subgraphs corresponding to the multiple image blocks from the second saliency map; calculate saliency weights of the multiple image blocks based on the saliency subgraphs corresponding to the multiple image blocks; and adjust the initial bit rates of the multiple image blocks using the saliency weights of the multiple image blocks, with the constraint that the sum of the bit rates of the multiple image blocks is equal to the target number of bits, to obtain target bit rates for the multiple image blocks.

[0177] Optionally, when calculating the saliency weights of multiple image blocks based on the saliency subgraphs corresponding to the multiple image blocks, the processor 80b is specifically configured to: calculate the sum of the saliency values ​​in the saliency subgraph corresponding to each image block; select a target image block having the largest sum of corresponding saliency values ​​from the multiple image blocks; and calculate, for any image block, the ratio between the sum of the saliency values ​​corresponding to any image block and the sum of the saliency values ​​corresponding to the target image block as the saliency weight of any image block.

[0178] In some embodiments, when the processor 80b performs image encoding on an e-commerce image based on a target bit rate of a plurality of image blocks, the processor 80b is specifically configured to: calculate image encoding parameters of the plurality of image blocks based on the target bit rates of the plurality of image blocks and a curve relationship between the bit rate and the image encoding parameters; and perform image encoding on the plurality of image blocks using the image encoding parameters of the plurality of image blocks to perform image encoding on the e-commerce image.

[0179] The computing device provided in this embodiment, when performing saliency detection on e-commerce images, takes into account that, for e-commerce images, users are more likely to be attracted to text in e-commerce images than to semantic objects; and that areas outside the text region still have a high degree of attention. Saliency detection and text detection are performed on the e-commerce images separately. A second saliency map for the e-commerce images is generated based on the first saliency map obtained by saliency detection and the text probability map obtained by text detection. This implements saliency detection tailored to the characteristics of the e-commerce images and helps improve the accuracy of saliency detection for e-commerce images.

[0180] In an embodiment of the present application, the processor 80b is further used to: obtain an image to be processed; the image to be processed includes: text information and images of other objects; perform saliency feature extraction on the image to be processed to obtain saliency features of the image to be processed; perform saliency detection on the image to be processed based on the saliency features to obtain a first saliency map of the image to be processed; perform text detection on the image to be processed based on the saliency features to obtain a text prediction probability map of the image to be processed; and generate a second saliency map of the image to be processed based on the first saliency map and the text prediction probability map.

[0181] Among them, regarding the specific implementation methods of performing text detection and saliency detection on the image to be processed, please refer to the relevant content of performing text detection and saliency detection on e-commerce images mentioned above, and will not be repeated here.

[0182] In some optional embodiments, such as Figure 8 As shown, the computing device may further include: a communication component 80c, a power supply component 80d, etc. In some embodiments, the computing device may be implemented as a terminal device such as a mobile phone or a computer, and accordingly, the computing device may further include: a display component 80e and an audio component 80f, etc. Figure 8Only some components are shown schematically, and it does not mean that the computing device must include Figure 8 The components shown do not necessarily mean that the computing device can only include Figure 8 Components shown.

[0183] In an embodiment of the present application, the memory is used to store computer programs and can be configured to store various other data to support operations on the device where it is located. The processor can execute the computer program stored in the memory to implement the corresponding control logic. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0184] In the embodiment of the present application, the processor can be any hardware processing device that can execute the logic of the above method. Optionally, the processor can be a central processing unit (CPU), a graphics processing unit (GPU) or a microcontroller unit (MCU); it can also be a programmable device such as a field programmable gate array (FPGA), a programmable array logic device (PAL), a general array logic device (GAL), a complex programmable logic device (CPLD); or an advanced reduced instruction set (RISC) processor (Advanced RISC Machines, ARM) or a system on chip (System on Chip, SOC), etc., but is not limited thereto.

[0185] In an embodiment of the present application, the communication component is configured to facilitate wired or wireless communication between the device in which it is located and other devices. The device in which the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G, 5G or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can also be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology or other technologies.

[0186] In an embodiment of the present application, the display component may include a liquid crystal display (LCD) and a touch panel (TP). If the display component includes a touch panel, the display component may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0187] In embodiments of the present application, a power supply assembly is configured to provide power to various components of the device in which it is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0188] In an embodiment of the present application, the audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as call mode, recording mode, and voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component further includes a speaker for outputting an audio signal. For example, for a device with a language interaction function, voice interaction with a user can be achieved through the audio component.

[0189] It should be noted that the descriptions of "first" and "second" in the inventor of this application are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit the "first" and "second" to different types.

[0190] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0191] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0192] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0193] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0194] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0195] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0196] Computer storage media is readable storage media, also known as readable media. Readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in the inventors of this application, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0197] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0198] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An e-commerce image processing method, characterized in that: include: Get e-commerce images; Extracting significant features of the e-commerce image to obtain significant features of the e-commerce image; performing saliency detection on the e-commerce image according to the saliency feature to obtain a first saliency map of the e-commerce image; Performing text detection on the e-commerce image based on the significant features to obtain a text probability map of the e-commerce image; the probability value in the text probability map refers to the probability that a pixel point in the e-commerce image is a text; A second saliency map of the e-commerce image is generated according to the first saliency map and the text probability map.

2. The method according to claim 1, characterized in that The extracting significant features of the e-commerce image to obtain the significant features of the e-commerce image includes: Extracting salient features of the e-commerce image using a backbone network of a saliency detection model to obtain salient features of the e-commerce image; The performing saliency detection on the e-commerce image according to the saliency feature to obtain a first saliency map of the e-commerce image includes: performing saliency detection on the e-commerce image according to the saliency features using the saliency detection head of the saliency detection model to obtain a first saliency map of the e-commerce image; The performing text detection on the e-commerce image based on the significant features to obtain a text probability map of the e-commerce image includes: The text detection head of the saliency detection model is used to perform text detection on the e-commerce image according to the saliency features to obtain a text probability map of the e-commerce image.

3. The method according to claim 2, characterized in that The text detection head using the saliency detection model performs text detection on the e-commerce image based on the saliency features to obtain a text probability map of the e-commerce image, including: Using the text detection head of the saliency detection model to perform character detection on the e-commerce image based on the saliency features, to obtain a character region score map of the e-commerce image; each score in the character region score map represents a probability that a corresponding pixel in the e-commerce image is the center of a character; Detecting the connection relationship between characters in the e-commerce image based on the significant features to obtain a character affinity score graph of the e-commerce image; the character affinity score represents the probability of adjacent character area centers; The character region score map and the character affinity score map are determined to be the text probability map.

4. The method according to claim 2, characterized in that Also includes: Get e-commerce image samples; Taking minimization of the loss function as the training objective, the backbone network, the saliency detection head, and the text detection head are trained using e-commerce image samples; During the model training process, multiple cascaded network layers in the backbone network are used to extract significant features of e-commerce image samples to obtain significant prediction features corresponding to the multiple network layers; Inputting the saliency prediction features corresponding to the multiple network layers into a text detection head to perform text detection, so as to obtain a text probability prediction map of the e-commerce image sample; Inputting the saliency prediction features corresponding to the last layer of the multiple network layers into a saliency detection head for performing saliency detection to obtain a saliency prediction map; The loss function is determined based on the difference between the text probability prediction map and the text probability true value map, the difference between the significance prediction map and the significance true value map, and the difference between the significance prediction features and the significance true value features corresponding to the multiple network layers; the significance true value features corresponding to each network layer are the same size as the significance prediction features corresponding to the network layer.

5. The method according to claim 4, characterized in that The model training process includes multiple training rounds, and the method further includes: For a current training round, inputting saliency prediction features corresponding to multiple network layers output by the backbone network in the current training round and a saliency prediction map output by the saliency detection head in the previous training round into the text detection head for text detection, thereby obtaining a text probability prediction map output by the text detection head in the current training round; Inputting the saliency prediction features corresponding to the last layer of the multiple network layers and the character probability prediction map output by the character detection head in the previous training round into the saliency detection head for saliency detection, so as to obtain a saliency prediction map output by the saliency detection head in the current training round; The current training round is any training round except the first training round.

6. The method according to claim 4, characterized in that The e-commerce image samples include: positive sample images and negative sample images; the positive sample images are e-commerce image samples with correct annotations on the known saliency true value map and the text probability true value map; the negative sample images are e-commerce image samples with incorrect annotations on the known saliency true value map and the text probability true value map.

7. The method according to any one of claims 1 to 6, characterized in that Generating a second saliency map of the e-commerce image according to the first saliency map and the text probability map includes: Pixel values ​​at the same pixel coordinates in the text probability map and the first saliency map are added to obtain a second saliency map of the e-commerce image.

8. The method according to any one of claims 1 to 6, characterized in that Also includes: Performing image segmentation on the e-commerce image to obtain a plurality of image blocks; performing bitrate allocation on the plurality of image blocks according to the second saliency map to obtain target bitrates for the plurality of image blocks; Image encoding is performed on the e-commerce image according to target bit rates of the multiple image blocks.

9. The method according to claim 8, characterized in that Also includes: Minimizing coding distortion of the e-commerce image is a goal, and the sum of the bit rates of the multiple image blocks is constrained to be equal to a set target number of bits, and using a rate-distortion optimization method to calculate the initial bit rates of the multiple image blocks; The performing bitrate allocation on the plurality of image blocks according to the second saliency map to obtain target bitrates for the plurality of image blocks includes: The initial bit rates of the plurality of image blocks are adjusted according to the second saliency map to obtain target bit rates of the plurality of image blocks.

10. The method according to claim 9, characterized in that The adjusting the initial bit rates of the plurality of image blocks according to the second saliency map includes: determining, from the second saliency map, saliency submaps corresponding to the plurality of image blocks; Calculating saliency weights of the multiple image blocks according to the saliency subgraphs corresponding to the multiple image blocks; With the sum of the bit rates of the multiple image blocks being equal to the target bit number as a constraint, the initial bit rates of the multiple image blocks are adjusted using the significance weights of the multiple image blocks to obtain the target bit rates of the multiple image blocks.

11. The method according to claim 8, characterized in that The performing image encoding on the e-commerce image according to the target bit rates of the multiple image blocks includes: Calculating image coding parameters for the plurality of image blocks according to target bit rates for the plurality of image blocks and a curve relationship between bit rates and image coding parameters; Image encoding is performed on the multiple image blocks using the image encoding parameters of the multiple image blocks, so as to perform image encoding on the e-commerce image.

12. An image processing method, characterized in that: include: Get the image to be processed; The images to be processed include: text information and images of other objects; Extracting significant features of the image to be processed to obtain significant features of the image to be processed; performing saliency detection on the image to be processed according to the saliency feature to obtain a first saliency map of the image to be processed; Performing text detection on the image to be processed based on the salient features to obtain a text probability map of the image to be processed; the probability value in the text probability map refers to the probability that a pixel point in the image to be processed is a text; A second saliency map of the image to be processed is generated according to the first saliency map and the text probability map.

13. A computing device, characterized in that include: A memory and a processor; wherein the memory is used to store a computer program; The processor is coupled to the memory and configured to execute the computer program to perform the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the steps of the method according to any one of claims 1 to 12.

Images