Image processing model acquisition, image processing method and device

Abstract

The application relates to the technical field of image processing, and discloses an image processing model acquisition method and device and an image processing method. In the method, for each sample image in a sample image set, a visual feature extraction network is used to extract the visual feature of the sample image; a text feature extraction network is used to extract the text feature of the sample image; a feature processing network is used to determine a first fusion feature and a first semantic matching score based on the visual feature and the text feature; a scoring network is used to determine an image representation score based on the first fusion feature and the first semantic matching score; and a first model is subjected to parameter adjustment based on a first loss function and a second loss function until a target image processing model is obtained. The above method is beneficial to improving the representation capability and accuracy of the image representation score output by the model.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing model acquisition, image processing method and apparatus. Background Technology

[0002] To achieve more efficient and convenient image matching, similarity calculations are often performed on the representational information of two images. In related technologies, different modal features extracted from images (such as visual features, text features, etc.) are often simply superimposed, and the image's representational information is determined based on this simple superposition result. This method of determining image representational information has limitations; it cannot effectively capture the correlation between different modal features, which can negatively impact the determination of image representational information, such as affecting its accuracy. Therefore, a more accurate and effective method for determining image representational information is needed. Summary of the Invention

[0003] To address at least one of the aforementioned technical problems, this application provides an image processing model acquisition method and apparatus. The technical solution of this application is as follows.

[0004] According to a first aspect of the embodiments of this application, an image processing model acquisition method is provided, the method comprising: For each sample image in the sample image set, a visual feature extraction network is used to extract the visual features of the sample image; Text features of the sample images are extracted using a text feature extraction network; A feature processing network is used to determine a first fusion feature and a first semantic matching score based on the visual features and the text features; the first semantic matching score measures the degree of semantic matching between the visual features and the text features. The scoring network is used to determine the image representation score based on the first fusion feature and the first semantic matching score; The parameters of the first model are adjusted based on the first loss function and the second loss function until the target image processing model is obtained; the first model includes the visual feature extraction network, the text feature extraction network, the feature processing network and the scoring network. The first loss function is constructed based on the difference between the first fused feature and the second fused feature of the reference image. The second loss function is constructed based on the difference between the first semantic matching score and the second semantic matching score of the reference image. The visual content similarity between the reference image and the sample image is greater than a first threshold and the text content similarity is less than a second threshold.

[0005] According to a second aspect of the embodiments of this application, an image processing method is provided, the method comprising: Obtain the image to be processed; The image representation score of the image to be processed is determined using the target image processing model obtained by training according to the image processing model acquisition method described in the first aspect.

[0006] According to a third aspect of the embodiments of this application, an image processing model acquisition apparatus is provided, the apparatus comprising: The visual feature extraction module is used to extract the visual features of each sample image in the sample image set using a visual feature extraction network. The text feature extraction module is used to extract text features from the sample image using a text feature extraction network; The feature processing module is used to determine a first fusion feature and a first semantic matching score based on the visual features and the text features using a feature processing network; the first semantic matching score measures the degree of semantic matching between the visual features and the text features. The scoring module is used to determine the image representation score based on the first fusion feature and the first semantic matching score using a scoring network; The parameter adjustment module is used to adjust the parameters of the first model based on a first loss function and a second loss function until a target image processing model is obtained. The first model includes the visual feature extraction network, the text feature extraction network, the feature processing network, and the scoring network. The first loss function is constructed based on the difference between the first fused feature and the second fused feature of the reference image. The second loss function is constructed based on the difference between the first semantic matching score and the second semantic matching score of the reference image. The visual content similarity between the reference image and the sample image is greater than a first threshold, and the text content similarity is less than a second threshold.

[0007] According to a fourth aspect of the embodiments of this application, an image processing apparatus is provided, the apparatus comprising: The image acquisition module is used to acquire the image to be processed. An image processing module is used to determine the image representation score of the image to be processed using a target image processing model trained according to the image processing model acquisition method described in the first aspect.

[0008] According to a fifth aspect of the embodiments of this application, a computing device is provided, the computing device including a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded and executed by the processor to implement the image processing model acquisition method as described in the first aspect, or the image processing method as described in the second aspect.

[0009] According to a sixth aspect of the embodiments of this application, a computer-readable storage medium is provided, which stores at least one computer program, which is loaded and executed by a processor to implement the image processing model acquisition method as described in the first aspect or the image processing method as described in the second aspect.

[0010] According to a seventh aspect of the present application, a computer program product is provided, the computer program product comprising at least one computer program, the at least one computer program being loaded and executed by a processor to implement the image processing model acquisition method as described in the first aspect, or the image processing method as described in the second aspect.

[0011] In an image processing model acquisition method provided in this application embodiment, during the training process to obtain the target image processing module, a first model including a visual feature extraction network, a text feature extraction network, a feature processing network, and a scoring network is used as the base model. The visual feature extraction network extracts visual features from the sample image, and the text feature extraction network extracts text features from the sample image. Then, the feature processing network determines a first fusion feature and a first semantic matching score based on the visual and text features. Furthermore, the scoring network determines an image representation score based on the fusion feature and the semantic matching score. Finally, the parameters of the first model are adjusted based on a first loss function and a second loss function until the target image processing model is obtained. This application embodiment, by introducing a first fusion feature and a first semantic matching score, generates an image representation score from both feature association and semantic matching dimensions, effectively improving the accuracy of image representation. In this application embodiment, the image representation score is used as the image representation information. Compared to the simple feature superposition results of visual and text features in related technologies, the generation of the image representation score relies on the first fusion feature, which is the fusion result of visual and text features, and also depends on the first semantic matching score, which measures the degree of semantic matching between visual and text features. With both feature association dimension (corresponding to the first fusion feature) and semantic matching dimension (corresponding to the first semantic matching score) as criteria, the representational ability of the image representation score can be improved. Simultaneously, using hard sample pairs to train the model can guide it to better distinguish images with similar visual content but different textual content, thus obtaining a more accurate and effective image representation capability. For the training of the first model, relying on the loss function of the fusion feature dimension and the loss function of the semantic matching score dimension provided by hard sample pairs (where the visual content similarity between the reference image and the sample image is greater than the first threshold, and the textual content similarity is less than the second threshold) helps guide the model's ability to distinguish features (information) between two images with "high visual content similarity but low textual content similarity" during training, thereby supporting the trained target image processing model to achieve accurate and effective image representation score output.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0013] Figure 1 This illustration shows an application environment diagram provided by an embodiment of this application; Figure 2 A flowchart of the image processing model acquisition method provided in an embodiment of this application is shown; Figure 3 A flowchart illustrating the fusion of textual and visual features provided in an embodiment of this application is shown; Figure 4 A schematic diagram of the first model provided in an embodiment of this application is shown; Figure 5 A schematic diagram of the difficult sample pairs provided in the embodiments of this application is shown; Figure 6 This illustration shows an application diagram of the target image processing model provided in an embodiment of this application; Figure 7 A flowchart of the image processing method provided in an embodiment of this application is shown; Figure 8 This paper shows a structural block diagram of the image processing model acquisition device provided in an embodiment of the present application; Figure 9 A structural block diagram of the image processing apparatus provided in an embodiment of this application is shown; Figure 10 A structural block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0016] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or at least two processors or memory) can be used to implement one or at least two modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0017] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0018] The term “exemplary” as used herein means “serving as an example or embodiment.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0019] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or at least two elements selected from the set consisting of A, B, and C. Additionally, "multiple" means two or more.

[0020] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed description. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0021] Figure 1 The illustration shows an application environment provided by an embodiment of this application. The application environment may include a terminal and a server. The terminal and the server can be directly or indirectly connected via wired or wireless communication.

[0022] The terminal can be a mobile phone, computer (such as a desktop computer, tablet computer, or laptop computer), augmented reality (AR) / virtual reality (VR) device, digital assistant, smart voice interaction device (such as a smart speaker), smart wearable device, smart home appliance, in-vehicle terminal, etc. It can run programs or web pages that support image processing model acquisition and / or image processing. The program supporting "image processing model acquisition" can be an application or app developed specifically for "image processing model acquisition" function, or it can be an application or app with "image processing model acquisition" functionality. Similarly, the program supporting "image processing" can be an application or app developed specifically for "image processing" function, or it can be an application or app with "image processing" functionality. In practical applications, this program supporting image processing model acquisition and / or image processing can be an AI assistant application, instant messaging application, news feed application, video application, social application, etc.

[0023] The server can be a standalone physical server, a server cluster or distributed system consisting of at least two physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server may include network communication units, processors, and memory.

[0024] In practical applications, the image processing model acquisition method provided in this application embodiment can be executed independently by the terminal, independently by the server, or collaboratively by the terminal and the server through interaction.

[0025] It should be noted that when sample images, reference images, and images to be processed that are related to user information are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0026] Figure 2 , 4 A flowchart illustrating an image processing model acquisition method provided in an embodiment of this application is shown. This method is used by an electronic device (such as...) Figure 1 Taking the terminal or server shown as an example, the method includes the following steps: Step 201: For each sample image in the sample image set, extract the visual features of the sample image using a visual feature extraction network; The sample image set may include multiple sample images.

[0027] The visual feature extraction network is responsible for encoding the visual content in the sample images. In practical applications, models such as ViT-Base and CLIP can be used for visual feature extraction networks.

[0028] The visual features of the sample images may include at least one of the following types: pixels, texture, grayscale, geometric shapes (such as outlines), color, and local details.

[0029] Among them, if the size of the sample image meets the input requirements of the visual feature extraction network (e.g., 224), then... If the sample image is 224 pixels, then input the sample image into the visual feature extraction network; otherwise, adjust the size of the sample image to meet the input requirements of the visual feature extraction network, and then input the resized sample image into the visual feature extraction network.

[0030] The visual feature extraction network can be pre-trained using ImageNet (a large visualization database).

[0031] Step 202: Extract text features from the sample image using a text feature extraction network.

[0032] The text feature extraction network is responsible for encoding the text content in the sample images. In practical applications, models such as BERT and RoBERTa-Base can be used for text feature extraction.

[0033] The text features of the sample images may include at least one of the following types: semantics, language, grammar, symbols, and sentiment.

[0034] In some embodiments, before extracting the text features of the sample image using a text feature extraction network, the method may further include the following steps: performing optical character recognition on the sample image to obtain first text.

[0035] Correspondingly, the step of extracting text features from the sample image using a text feature extraction network may include the following steps: extracting features from the first text using the text feature extraction network to obtain the text features.

[0036] Furthermore, before extracting features from the first text using the text feature extraction network, the first text can be cleaned first, and then the cleaned first text can be input into the text feature extraction network. Data cleaning focuses on identifying and correcting dirty data. Data cleaning may involve at least one of the following: completing missing text segments, correcting erroneous text segments, deleting duplicate text segments, processing (deleting, replacing, retaining) abnormal text segments, standardizing text format, and performing word segmentation on the text.

[0037] In one specific embodiment, the OCR unit and the text feature extraction network can constitute a text content processing module for encoding the text content in the sample image. Specifically, the OCR unit is responsible for performing optical character recognition on the sample image to obtain the first text, and the text feature extraction network is responsible for extracting text features from the first text.

[0038] In some embodiments, before extracting the text features of the sample image using a text feature extraction network, the method may further include the following steps: first, performing optical character recognition on the sample image to obtain first text; then, using a large language model to correct the first text based on a preset prompt text to obtain second text.

[0039] Correspondingly, the step of extracting text features from the sample image using a text feature extraction network may include the following steps: extracting features from the second text using the text feature extraction network to obtain the text features.

[0040] Among them, the use of Optical Character Recognition (OCR) technology can extract information such as text, logos, watermarks, and annotations from sample images, realizing the conversion of uneditable image text content into computer-processable first text.

[0041] Among them, large language models can adopt open-source LLaMA-7B models, GPT models, etc.

[0042] The large language model's correction of the first text aims to improve its readability and quality. The large language model performs text correction through at least one of the following: correcting garbled characters, correcting typos, simplifying language (e.g., replacing complex, abstract terms with simple, concrete vocabulary), shortening sentences (breaking long, complex sentences into multiple shorter sentences), and optimizing structure (e.g., using headings, lists, and paragraphs to organize content for clearer logic).

[0043] The preset prompt text is used to guide the large language model to correct the first text. Furthermore, the preset prompt text can carry scene prompts, which can guide the large language model to understand and correct the first text based on a specific scene, and can also constrain the large language model to adapt the strength of text correction to the specific scene.

[0044] Optical character recognition (OCR) technology can improve the efficiency and convenience of text information extraction. Combined with text correction and the natural language processing capabilities of large language models, the readability and quality of the input text feature extraction network can be improved. This helps reduce text feature inaccuracies caused by text recognition biases (e.g., using large language models to correct text biases caused by limited OCR capabilities (e.g., blurry text), thus avoiding semantic inaccuracies in the extracted text features). It also helps improve the accuracy of text feature extraction based on more readable and higher-quality text (e.g., using large language models to simplify special identifiers obtained from OCR, making them easier for the text feature extraction network to understand).

[0045] Furthermore, before using a large language model to correct the first text based on preset prompts to obtain the second text, the first text can be cleaned first, and then the cleaned first text can be input into the large language model. Data cleaning focuses on identifying and correcting dirty data. Data cleaning may involve at least one of the following: completing missing text segments, correcting erroneous text segments, deleting duplicate text segments, processing (deleting, replacing, retaining) abnormal text segments, standardizing text format, and performing word segmentation on the text.

[0046] In one specific embodiment, an OCR unit, a large language model, and a text feature extraction network can constitute a text content processing module for encoding the text content in a sample image. Specifically, the OCR unit performs optical character recognition on the sample image to obtain the first text, the large language model corrects the first text based on a preset prompt to obtain the second text, and the text feature extraction network extracts text features from the second text. When subsequently adjusting the parameters of the text content processing module, only the parameters of the large language model and the text feature extraction network need to be adjusted. Low-rank adaptation techniques can be used when adjusting the parameters of the large language model.

[0047] In practical applications, as business scenarios change (such as adding new abnormal image categories or changes in the original abnormal image category identification standards), targeted prompt text can be designed to guide the correction direction of the large language model.

[0048] After training and obtaining the target image processing model, the large language model can also be fine-tuned and upgraded based on the newly added sample images.

[0049] Step 203: Utilize a feature processing network to determine a first fusion feature and a first semantic matching score based on the visual features and the text features; the first semantic matching score measures the degree of semantic matching between the visual features and the text features.

[0050] On one hand, the feature processing network is responsible for deep fusion of visual and textual features to generate the first fused feature. This maps visual and textual features to the same feature space, deeply exploring the intrinsic relationships between the two types of features and achieving deep fusion of features from different modalities rather than simple feature superposition. On the other hand, the feature processing network is responsible for measuring the semantic matching degree of visual and textual features to generate a first semantic matching score.

[0051] The first semantic matching score serves as a quantitative indicator, and its value range can be [0, 1], [0, 100], [0, 1000], etc. A higher first semantic matching score indicates a smaller semantic difference between the visual content and text content within the sample image; a lower first semantic matching score indicates a larger semantic difference between the visual content and text content within the sample image. Taking the first semantic matching score range of [0, 1] as an example, the semantic matching score threshold can be 0.8. If it is greater than or equal to 0.8, it indicates a high semantic matching within the image.

[0052] The feature processing network can employ a large language model, such as the open-source LLaMA-7B model or the GPT model. Subsequent parameter adjustments to the feature processing network can utilize low-rank adaptation techniques.

[0053] The first fusion feature and the first semantic matching score can be further fused into a comprehensive feature. Correspondingly, the subsequent scoring network can determine the image representation score based on the comprehensive feature. For example, the first fusion feature and the first semantic matching score are concatenated and mapped to a specific dimension (e.g., 768) through the fully connected output layer of the feature processing network to obtain the comprehensive feature. LayerNorm (layer normalization) is used in the output layer to ensure a stable feature distribution.

[0054] In some embodiments, such as Figure 3As shown, the visual features include global visual features and multiple first local visual features, and the text features include global text features and multiple first local text features. The step of using a feature processing network to determine the first fusion feature based on the visual features and the text features includes: Step 301: Use the feature processing network to fuse the global visual features and the global text features to obtain the global fused features; Step 302: For each of the first local text features, update the first local text features based on the text weights of the first local text features using the feature processing network to obtain the corresponding second local text features; Step 303: For each first local visual feature, the feature processing network is used to obtain the corresponding second local visual feature based on the text weight of each first local text feature and the attention weight between the first local visual feature and each first local text feature. Step 304: Use the feature processing network to fuse multiple second local visual features and multiple second local text features to obtain local fused features; Step 305: Using the feature processing network, the first fusion feature is obtained based on the global fusion feature and the local fusion feature.

[0055] Global visual features can be obtained by encoding the visual content of the sample image as a whole. In contrast, local visual features can be obtained by encoding a specific part of the visual content. For example, a 224-bit image... A 224-pixel sample image is input into the visual feature extraction network, which outputs a 768-dimensional global visual feature and 256 768-dimensional local visual features. The 256 local visual features correspond one-to-one with 256 local visual regions.

[0056] Global text features can be obtained by encoding the entire text content in the sample image. In contrast, local text features can be obtained by encoding a specific part of the text content. For example, if the text content in the sample image indicates N words, the visual feature extraction network outputs a 768-dimensional global text feature and N 768-dimensional local text features, with each of the N local text features corresponding one-to-one with the N words.

[0057] Step 301 focuses on the fusion of global visual features and global text features, resulting in a global fused feature. Steps 302-304 focus on the fusion of multiple first local visual features and multiple first local text features, resulting in a local fused feature. Step 305 focuses on the fusion of global fused features and local fused features, resulting in a first fused feature.

[0058] The fusion of global visual features and local image features can be achieved through a cross-attention mechanism.

[0059] In the process of fusing multiple first local visual features and multiple first local text features, each first local text feature is updated from the perspective of text weight. For each first local visual feature, the update considers the attention weights between it and each first local text feature, as well as the text weights of each first local text feature itself. The attention weights between each first local visual feature and each first local text feature can be determined using a cross-attention mechanism.

[0060] In cases where the dimensions of multiple first local visual features and multiple first local text features are large (e.g., 786), a linear projection layer can be used to uniformly map all local features to an intermediate space with smaller dimensions (e.g., 512), thereby reducing the amount of computation.

[0061] The fusion between global and local fusion features can be achieved through feature concatenation.

[0062] The composition of visual and textual features was set with finer granularity, providing more informative visual and textual features that can fully capture the correlations between different modal features. Simultaneously, the orderly and hierarchical fusion of visual and textual features can maximize the expressive power of different features and uncover the correlations between them. This is beneficial for improving the representational ability of the obtained first fused feature.

[0063] For example, global visual features can be... and global text features Inputting the first fully connected network (with 1024 hidden layers and ReLU activation) yields globally fused features. .

[0064] Multiple first-local visual features can be viewed as a single local visual feature matrix V. Let represent the j-th first local visual feature. Multiple first local text features can be viewed as a single local text feature matrix T. Let V represent the i-th local text feature. By performing cross-attention processing on the local visual feature matrix V and the local text feature matrix T, we can obtain the cross-attention matrix. , = Where N represents the total number of text features in the first local area, M represents the total number of visual features in the first local area, and d represents the number of dimensions.

[0065] Correspondingly, multiple second local visual features can be represented as the following matrix = .in, This represents the text weight of the i-th first local text feature. Multiple second local text features can be represented by the following matrix. =

[0066] Based on this, it is possible to and Inputting a second fully connected network (with 1024 hidden layers and ReLU activation) yields locally fused features. .

[0067] In a further embodiment, the text weight of the first local text feature can be determined by the following steps: The text weight of the first local text feature is determined based on whether the local text is located within a visually salient region of the sample image; the local text is used to extract the first local text feature; or... The text weight of the first local text feature is determined based on the importance of the local text; or, The first weight of the first local text feature is determined based on whether the local text is located in the visually salient region of the sample image; the second weight of the first local text feature is determined based on the importance of the local text; and the text weight of the first local text feature is obtained based on the first weight and the second weight.

[0068] The granularity of local text can be words, phrases, etc. In contrast, global text refers to all the text used by the text feature extraction network to output text features.

[0069] The importance of a local text can be determined by the frequency of its occurrence in the entire text, or it can be based on the term frequency-inverse document frequency (TF-IDF) of the local text. In this case, the entire text is equivalent to a file, and the file set can be determined based on the sample image set (e.g., one sample image in the sample image set is used to provide one file in the file set).

[0070] Visually salient regions can be determined through visual saliency detection of sample images (using intelligent algorithms to simulate human visual characteristics and extract visually salient regions from the image; that is, regions of interest to humans). Visually salient regions are the primary visual areas in the sample image, in contrast to the background areas.

[0071] This section provides three methods for determining text weights, offering flexible options for determining text weights based on comprehensive efficiency and accuracy requirements in practical applications. One method determines text weights based on whether the local text is located within a visually salient region of the sample image, taking into account the varying correlations between the location of the local text and visual features, thus achieving dynamic text weight allocation based on visual region perception. Another method determines text weights based on the importance of the local text, taking into account whether the local text possesses semantic importance.

[0072] For example, the sample image contains M local visual regions (patches) of the same size. A visual saliency detection algorithm is used to identify visually saliency regions (the set of local visual regions with the top 30% saliency scores) from these M local visual regions. ) and background area (the remaining set of local visual areas) The coordinate information of multiple local texts is output using OCR technology. For each local text coordinate, if it is located within a visually salient region, the first weight is set to 1.5; if it is located within a background region, the first weight is set to 0.8. The second weight of each local text is determined by calculating the term frequency-inverse document frequency. The text weight can be the product of the first and second weights.

[0073] Considering that a local text may simultaneously occupy at least one visually salient region and at least one background region, a corresponding weight set can be determined based on the actual region occupied by the local text. The weight set includes a preset weight corresponding to each region actually occupied by the local text; for example, the preset weight for a visually salient region is 1.5, and the preset weight for a background region is 0.8. A multiplication operation is performed on each weight in the weight set to obtain the first weight.

[0074] In addition, the global fusion features and local fusion features can be concatenated and then input into a binary classification subnetwork (two fully connected layers, output dimension 1, sigmoid activation). This subnetwork outputs the first semantic matching score.

[0075] In some embodiments, the feature processing network can be obtained through the following steps: First, for each first image in the candidate image set, the visual features and text features corresponding to the first image are fused using a preset network to obtain the fused features of the first image; then, a fourth loss function is constructed based on the difference between the fused features of the first image and the fused features of the second image; the visual content similarity between the second image and the first image is greater than a third threshold, and the text content similarity is less than a fourth threshold; furthermore, the parameters of the preset network are adjusted based on the fourth loss function until the feature processing network is obtained.

[0076] The candidate image set may include multiple first images.

[0077] The visual content similarity between the first and second images is greater than a third threshold, indicating high visual similarity between them. The third threshold can be an empirical value and can be flexibly adjusted based on the business scenario. In practical applications, the third threshold can be the same as the first threshold. The text content similarity between the first and second images is less than a second threshold, indicating low text content similarity between them. The fourth threshold can also be an empirical value and can be flexibly adjusted based on the business scenario. In practical applications, the fourth threshold can be the same as the second threshold. The first and second images can be considered a hard sample pair. The second image can be taken from a candidate image set or generated by at least one text content transformation operation on the first image. At least one text content transformation operation includes at least one of the following: a local replacement operation on the original text content, a local deletion operation on the original text content, a partial occlusion operation on the original text content, a blurring operation on the original text content, or an operation indicating the addition of preset text to the original image. In practical applications, the text content affected by the text content transformation operation can be located in a visually salient region of the original image. The text content transformation operation does not affect the visual region of the original image.

[0078] The difference between two fused features can be calculated by measuring Hamming distance, Euclidean distance, cosine similarity, etc.

[0079] Guided by the fourth loss function, the network can be instructed to adjust its parameters to increase the difference between the two fused features. During the adjustment of network parameters, the fused features of the image may change, and correspondingly, the difference in fused features between the hard sample pairs may change. These changes can be reflected to some extent by the value of the fourth loss function.

[0080] It should be noted that adjusting the parameters of the preset network can involve at least one iteration. This means that, using a candidate image set as the training data source, and training the feature processing network using the preset network as the base network, at least one network parameter adjustment can be involved. Each network parameter adjustment is based on the fourth loss function corresponding to each first image in a subset of candidate images in one iteration. A subset of candidate images in one iteration can refer to all first images in the candidate image set, or it can refer to a portion of the first images in the candidate image set.

[0081] The feature processing network is a pre-trained result, possessing certain feature fusion capabilities and the ability to distinguish between two images with "high visual similarity but low textual similarity." Based on this, incorporating the feature processing network into the first model to be trained can improve the efficiency and convenience of training the target image processing model based on the first model.

[0082] For example, the fourth loss function can be expressed as ,in This represents the cosine similarity between the fused features of the first image and the fused features of the second image. This represents the similarity threshold, which can be set to 0.3 to determine if the similarity of the fused features of difficult sample pairs is below this threshold.

[0083] Step 204: Use a scoring network to determine the image representation score based on the first fusion feature and the first semantic matching score.

[0084] The scoring network is responsible for generating an image representation score based on the first fusion feature and the first semantic matching score. For example: 1) A basic representation score can be generated based on the first fusion feature; then, the product of the first semantic matching score and the basic representation score is taken as the image representation score. 2) A basic representation score can be generated based on the first fusion feature; then, the basic representation score is corrected according to the correction coefficient corresponding to the first semantic matching score to obtain the image representation score. The correction coefficient corresponding to the first semantic matching score is determined as follows: a) A target score interval containing the first semantic matching score can be determined from multiple preset score intervals; the preset correction coefficient carried by the target score interval is determined as the correction coefficient corresponding to the first semantic matching score. Multiple preset score intervals do not overlap, and the union of multiple preset score intervals covers continuous preset score segments. b) Alternatively, the product of the first semantic matching score and the adjustment coefficient can be taken as the correction coefficient corresponding to the first semantic matching score. The adjustment coefficient can be flexibly set according to actual business needs, aiming to avoid the overall image representation score being too large or too small, or having too many or too few digits.

[0085] Among them, the image representation score is a quantitative indicator, and its value range can be [0, 1], [0, 100], [0, 1000], etc.

[0086] In practical applications, the image representation score determination logic of the scoring network can differ for different business scenarios. For example, in a copyright scenario, the copyright identifier in an image corresponds to a score weight contribution. As the business scenario changes, the image representation score determination logic of the scoring network can be adjusted individually. Accordingly, the scoring network can select the corresponding image representation score determination logic based on the image category label output by the feature processing network (refer to the relevant description in step 205 below, which will not be repeated here).

[0087] Step 205: Adjust the parameters of the first model based on the first loss function and the second loss function until the target image processing model is obtained; the first model includes the visual feature extraction network, the text feature extraction network, the feature processing network and the scoring network, the first loss function is constructed based on the difference between the first fused feature and the second fused feature of the reference image, the second loss function is constructed based on the difference between the first semantic matching score and the second semantic matching score of the reference image, the visual content similarity between the reference image and the sample image is greater than the first threshold and the text content similarity is less than the second threshold.

[0088] In this context, a visual content similarity between the reference image and the sample image exceeding a first threshold indicates a high degree of visual similarity between them. The first threshold can be an empirical value and can be flexibly adjusted based on the business scenario.

[0089] If the text content similarity between the reference image and the sample image is less than the second threshold, it indicates low text content similarity between the two images. The second threshold can be an empirical value and can be flexibly adjusted according to the business scenario. The reference image and the sample image can be considered as a difficult sample pair. The reference image can be taken from the sample image set or generated by performing at least one text content transformation operation on the sample image. At least one text content transformation operation includes at least one of the following: a local replacement operation on the original text content, a local deletion operation on the original text content, a partial occlusion operation on the original text content, a blurring operation on the original text content, or an operation indicating the addition of preset text to the original image. In practical applications, the text content affected by the text content transformation operation can be located in the visually salient region of the original image. The text content transformation operation does not affect the visual region of the original image. The determination of the second fusion feature of the reference image can refer to the determination of the first fusion feature of the sample image, and will not be repeated here. The determination of the second semantic matching score of the reference image can refer to the determination of the first semantic matching score of the sample image, and will not be repeated here.

[0090] The difference between two fused features can be calculated by measuring Hamming distance, Euclidean distance, cosine similarity, etc.

[0091] The difference between two semantic matching scores can be calculated by measuring the difference, Hamming distance, etc.

[0092] The first model can be obtained by pre-training using a general image set.

[0093] Guided by the two types of loss functions, the model can be adjusted to increase the difference between the two fused features and the difference between the two semantic matching scores. During the adjustment of model parameters, the fused features of the image may change, the semantic matching score of the image may change, and correspondingly, the difference in fused features corresponding to hard sample pairs may change, and the semantic matching score corresponding to hard sample pairs may change. These changes can be reflected to some extent by the values ​​of the two types of loss functions.

[0094] It should be noted that parameter tuning of the first model can involve at least one iteration. This means that, using the sample image set as the training data source, in the process of training the target image processing model based on the first model, at least one model parameter tuning may be involved. Each model parameter tuning is based on the two-class loss function corresponding to each sample image in a subset of sample images in one iteration. A subset of sample images in one iteration can refer to all sample images in the sample image set, or it can refer to a portion of the sample images in the sample image set.

[0095] In some embodiments, adjusting the parameters of the first model based on the first loss function and the second loss function until the target image processing model is obtained may include the following steps: adjusting the parameters of the second model based on the first loss function and the second loss function until the target image processing model is obtained; the target image processing model includes the visual feature extraction network and the parameter-adjusted second model, the second model including the text feature extraction network, the feature processing network and the scoring network.

[0096] Freezing all parameters of the visual feature extraction network, preventing it from participating in multimodal training and using it solely as a visual extraction tool, reduces the number of training parameters and ensures the stability of visual feature extraction. Conversely, using a text feature extraction network, a feature processing network, and a scoring network to obtain the target image processing model balances training efficiency and effectiveness while reducing training costs.

[0097] In some embodiments, the feature processing network can also be used to determine image category labels based on the visual features and the text features.

[0098] Correspondingly, the step of adjusting the parameters of the first model based on the first loss function and the second loss function until the target image processing model is obtained may include the following steps: adjusting the parameters of the first model based on the first loss function, the second loss function and the third loss function until the target image processing model is obtained; the third loss function is constructed based on the difference between the image category label and the image category information carried by the sample image.

[0099] The introduction of the third loss function aims to guide the model to adjust its parameters to reduce the difference between the actual predicted image category label and the ideal image category information. This helps guide the model to learn image classification during training, enabling more flexible and adaptive processing of related images based on image category labels. For example, it stops matching operations on related images carrying anomalous labels to avoid affecting the stability of image matching.

[0100] In practical applications, image category labels can be used to locate the business scenario to which an image is applicable. These labels, along with fused features and semantic matching scores, serve as the basis for the scoring network to determine the image representation score. As the business scenario changes (such as adding new image categories or changes in the original image category identification standards), the logic for determining the image category labels in the feature processing network can be adjusted separately.

[0101] In a specific embodiment, the parameters of the second model can be adjusted based on the first loss function, the second loss function, and the third loss function until the target image processing model is obtained; the target image processing model includes a visual feature extraction network and the parameter-adjusted second model, and the second model includes a text feature extraction network, a feature processing network, and a scoring network.

[0102] For example, the first loss function, the second loss function, and the third loss function can constitute a combined loss function, such as... .in, This represents the cross-entropy loss (corresponding to the third loss function, used to optimize label classification accuracy). This represents the cosine similarity loss (corresponding to the first loss function, used to optimize the accuracy of similarity determination). This represents the image-text semantic contrast loss (corresponding to the second loss function, used to optimize the discrimination accuracy of hard sample pairs). , , .

[0103] The training can be set to 30 iterations with an initial learning rate of 5e-5. The learning rate decays to 0.5 every 10 iterations. The AdamW optimizer can be used, with a batch size of 32. During training, the parameters of each network are continuously adjusted using backpropagation to reduce training loss, enabling the model to accurately extract text features, achieve deep fusion of different modal features, and output reasonable image representation scores.

[0104] In addition, 1) Regarding data preparation: a large amount of labeled training data was collected, covering the target business scenarios; the training data was preprocessed: in terms of the visual content of the images, size normalization (e.g., 224) was performed. 2) For image text content, data cleaning and encoding are performed to convert it into text data that can be processed by the model. The preprocessed training data is divided into training set, validation set and test set in a ratio of 8:1:1 for model training, validation and testing. 2) For model validation and tuning, the trained model is used on the validation set to calculate core indicators such as accuracy, false positive rate, recall rate and false positive rate in scenarios with "high visual similarity but low text similarity". According to the validation results, the label weights (abnormal label weight is set to 1.2 and normal label weight is set to 1.0), the parameters of the scoring network and the semantic matching score threshold (initial threshold is set to 0.8) are dynamically adjusted to optimize model performance. If the validation set accuracy does not reach 98.5% or the false positive rate in a specific scenario is not lower than 2%, the training parameters are adjusted (such as increasing epochs, adjusting the learning rate, and supplementing hard sample pairs) and retrained until the model indicators meet the preset requirements. 3) In terms of model testing and deployment, the optimized model is used in the test set to test its judgment performance under different scenarios and different types of images, ensuring that the model can still maintain high accuracy in complex scenarios (such as image occlusion, text blurring, slight tampering, "high similarity of visual content, but low similarity of text content"). After the test is passed, the model is deployed to the production environment and integrated with the business system to support high-concurrency request processing.

[0105] by Figure 5 Taking the difficult sample pair as an example, the two images have identical visual content (both depict a person wearing a black T-shirt standing against a white background), but the text on the T-shirts differs: one image displays "sprinkling water," while the other displays "alcohol." Visual saliency detection identifies the black T-shirt area as a visually salient region. The coordinate information identified by OCR technology indicates that the T-shirt text in both images lies within this salient visual region, with a first weight of 1.5. TF-IDF calculations show that both images contain words with high semantic importance, with a second weight of 1.2. The semantic matching scores for the two images are 0.92 and 0.88, respectively. The image representation scores for the two images are 0.95 and 0.68, respectively. The score difference of 0.27 exceeds the threshold of 0.2, indicating dissimilarity and successfully avoiding misjudgment.

[0106] As can be seen from the technical solutions provided in the embodiments of this application above, in the process of training the target image processing module, a first model including a visual feature extraction network, a text feature extraction network, a feature processing network, and a scoring network is used as the base model. The visual feature extraction network extracts visual features from the sample image, and the text feature extraction network extracts text features from the sample image. Then, the feature processing network determines a first fusion feature and a first semantic matching score based on the visual and text features. Furthermore, the scoring network determines an image representation score based on the fusion feature and the semantic matching score. Finally, the parameters of the first model are adjusted based on a first loss function and a second loss function until the target image processing model is obtained. This embodiment of the application, by introducing a first fusion feature and a first semantic matching score, generates an image representation score from both feature association and semantic matching dimensions, effectively improving the accuracy of image representation. In this embodiment, the image representation score is used as the image representation information. Compared to the simple feature superposition results of visual and text features in related technologies, the generation of the image representation score relies on the first fusion feature, which is the fusion result of visual and text features, and also depends on the first semantic matching score, which measures the degree of semantic matching between visual and text features. With both feature association dimension (corresponding to the first fusion feature) and semantic matching dimension (corresponding to the first semantic matching score) as criteria, the representational ability of the image representation score can be improved. Simultaneously, using hard sample pairs to train the model can guide it to better distinguish images with similar visual content but different textual content, thus obtaining a more accurate and effective image representation capability. For the training of the first model, relying on the loss function of the fusion feature dimension and the loss function of the semantic matching score dimension provided by hard sample pairs (where the visual content similarity between the reference image and the sample image is greater than the first threshold, and the textual content similarity is less than the second threshold) helps guide the model's ability to distinguish features (information) between two images with "high visual content similarity but low textual content similarity" during training, thereby supporting the trained target image processing model to achieve accurate and effective image representation score output.

[0107] Figure 6 , 7 A flowchart illustrating an image processing method provided in an embodiment of this application is shown. This method is used by an electronic device (such as...) Figure 1 Taking the terminal or server shown as an example, the method includes the following steps: Step 701: Obtain the image to be processed; Step 702: Determine the image representation score of the image to be processed using the target image processing model obtained by training according to the image processing model acquisition method described in steps 201-205.

[0108] The image to be processed can be provided by the user, such as uploaded by the user through an interactive interface on an electronic device. Alternatively, the image can be provided by the business system, such as when the business system selects an image from its background image pool as the image to be processed.

[0109] In some embodiments, the similarity of two images can be determined by comparing the difference (absolute difference) between their respective image representation scores with a preset difference. If the difference is greater than or equal to the preset difference, the two images are not similar. If the difference is less than the preset difference, the two images are similar. For example, the image representation score can range from [0, 1], and the preset difference can be 0.2. Furthermore, the preset difference can be flexibly adjusted according to the business scenario. For example, the preset difference is 0.8 in a content review scenario and 0.75 in a copyright scenario.

[0110] In one specific embodiment, the similarity between two images is determined by comparing the difference (absolute value difference) between the image representation score of the image to be processed and the image representation score of the reference image with a preset difference. If the reference image is an abnormal image in a content review scenario (such as an image that violates relevant platform requirements), and the two images are similar, it indicates that the image to be processed is also an abnormal image. If the reference image is a copyrighted image in a copyright scenario, and the two images are similar, it indicates that the image to be processed may be an infringing image. If the reference image is a first-release image in an information filtering scenario, and the two images are similar, it indicates that the image to be processed may be a duplicate image. If the reference image is an introductory image of a product in a product monitoring scenario, and the two images are similar, it indicates that the image to be processed may be an introductory image of the same product.

[0111] In some embodiments, referring to the description of step 202 above, before the text features of the image to be processed are extracted by the target image processing model, they can first undergo optical character recognition, and then the target image processing model extracts the text features of the image to be processed based on the optical character recognition results; alternatively, they can first undergo optical character recognition, and then the large language model corrects the optical character recognition results, and then the target image processing model extracts the text features of the image to be processed based on the corrected results.

[0112] In some embodiments, if the visual feature similarity between the image to be processed and the reference image is greater than a fifth threshold, a third semantic matching score for the image to be processed and the reference image can be determined using the target image processing model. The third semantic matching score is determined based on the text features of the image to be processed and the visual features of the reference image, or the visual features of the image to be processed and the text features of the reference image. Then, if the third semantic matching score is less than a sixth threshold, the reference difference is updated based on a first magnification factor to obtain a target difference. The reference difference is determined based on the difference between the image representation score of the image to be processed and the image representation score of the reference image. Furthermore, if the target difference is less than a preset difference, the image to be processed is determined to be a similar image to the reference image.

[0113] The fifth threshold can be an empirical value. It can be flexibly adjusted based on the business scenario.

[0114] The sixth threshold can be an empirical value. It can be flexibly adjusted according to the business scenario. For example, if the semantic matching score range is [0, 1], the sixth threshold can be 0.8. If it is greater than or equal to 0.8, it indicates a high semantic matching across images.

[0115] The image to be processed can be an image that requests image matching with a reference image. It can be understood that there are multiple candidate images, and a reference image is determined from these candidate images. If the reference image is determined, image matching is performed between the reference image and the image to be processed. Alternatively, if the image to be processed is determined, a reference image is determined from multiple candidate images based on the image category of the image to be processed and / or the current business scenario, and then image matching is performed between the reference image and the image to be processed.

[0116] The reference image, also known as the seed image, can be an image that has already been processed by the target image processing model. Relevant information about the reference image (such as text features, visual features, fusion features, semantic matching scores, and image representation scores) can be stored and maintained (e.g., through table storage). This avoids repeated processing of the reference image using the target image processing model, thereby reducing computational resource consumption and improving business processing efficiency. In practical applications, candidate images and their related information can be dynamically updated according to business scenarios (such as content review scenarios, copyright scenarios, etc.). Multiple candidate images will serve as the data source for the reference image. For example, after image i is initially identified as a candidate image for scenario j, its relevant information is stored. If scenario j is adjusted, and image i is no longer identified as a candidate image for scenario j, its relevant information is deleted. Of course, as the model is updated, the stored and maintained information about the candidate images is also updated in a timely manner.

[0117] The third semantic matching score can be generated by the network responsible for generating semantic matching scores in the target image processing model.

[0118] To address the issue of high visual similarity between the image to be processed and the reference image, a cross-image, cross-modal semantic consistency verification mechanism is introduced. If the cross-image, cross-modal semantic matching score is low, the difference between the image representation scores of the two images is amplified, increasing the difficulty for them to fall into the range smaller than a preset difference. In this way, even if the visual features of two images are highly similar, they can still be accurately distinguished despite significant cross-image, cross-modal semantic differences.

[0119] For example, the image representation score can range from [0, 1], and the semantic matching score can also range from [0, 1]. The sixth threshold can be 0.3, and the first amplification factor can be 2.

[0120] In some embodiments, if the image category label of the image to be processed is an anomalous label, the image representation score of the image to be processed can be updated based on a second magnification factor to obtain a target score; then, if the difference between the target score and the image representation score of the reference image is less than a preset difference, the image to be processed is determined to be a similar image to the reference image.

[0121] Normally, if two images are similar, the difference between their respective image representation scores will also be close, falling within a range smaller than the preset difference. However, if one image carries an anomaly label, under the influence of the second magnification factor, the difference between the image representation scores of the two images will increase, thus increasing the difficulty for them to fall within the range smaller than the preset difference. This can, to some extent, avoid judging normal images and abnormal images as similar, thereby affecting the user experience.

[0122] For example, the image representation score can range from [0, 1], and the semantic matching score can also range from [0, 1]. The second amplification factor mentioned above can be 1.2.

[0123] In practical applications, image category tags can be used to pinpoint the appropriate business scenario for an image. For example, an "abnormal" tag in a content moderation scenario indicates that the image to be processed may violate the relevant platform requirements. An "abnormal" tag in a copyright scenario indicates that the image to be processed may be an infringing image.

[0124] In one specific embodiment, the image processing method provided in this application can achieve accurate image similarity determination. Multiple candidate images will serve as the data source for the reference image. Candidate images can be provided to the business system individually or in batches by the user, or selected individually or in batches by the business system. For the candidate images, the business system performs size normalization (224). The image undergoes preprocessing such as 224-pixel removal, noise removal, and blur repair. Then, the target image processing model is used to process the preprocessed candidate image. The final output and intermediate output of the model are stored and maintained as relevant information of the candidate image (such as fusion features, semantic matching scores, image representation scores, etc.).

[0125] When the business system selects an image to be processed or receives an image to be processed provided by the user, it can normalize the size of the image to be processed (224). The image undergoes preprocessing such as 224-pixel noise removal and blur restoration. Then, the target image processing model is used to process the preprocessed image. The final output and intermediate output of the model are used as relevant information of the image to be processed (such as text features, visual features, fusion features, semantic matching score, image representation score, etc.).

[0126] Based on this, 1) when the visual feature similarity between the image to be processed and the reference image is less than or equal to the fifth threshold, and the image category label of the image to be processed is a normal label, the similarity between the two images can be determined by comparing the difference (absolute value difference) between their respective image representation scores with the preset difference. If the difference is greater than or equal to the preset difference, the two images are not similar. If the difference is less than the preset difference, the two images are similar.

[0127] 2) If the visual feature similarity between the image to be processed and the reference image is less than or equal to the fifth threshold, and the image category label of the image to be processed is an abnormal label, the image representation score of the image to be processed can be updated based on the second magnification factor to obtain the target score. Then, the difference between the target score and the image representation score of the reference image (absolute value difference) is compared with the preset difference to determine whether the two images are similar.

[0128] 3) When the visual feature similarity between the image to be processed and the reference image is greater than the fifth threshold, and the image category label of the image to be processed is a normal label, the target image processing model can be used to determine the third semantic matching score between the image to be processed and the reference image. The third semantic matching score is determined based on the text features of the image to be processed and the visual features of the reference image, or vice versa. Then, when the third semantic matching score is less than the sixth threshold, the reference difference is updated based on the first amplification factor to obtain the target difference. The reference difference is determined based on the difference (absolute value difference) between the image representation score of the image to be processed and the image representation score of the reference image. Furthermore, the comparison result between the target difference and the preset difference is used to determine whether the two images are similar. Of course, if the third semantic matching score is greater than or equal to the sixth threshold, the comparison result between the difference (absolute value difference) between the image representation scores of the two images and the preset difference is used to determine whether the two images are similar.

[0129] 4) When the visual feature similarity between the image to be processed and the reference image is greater than the fifth threshold, and the image category label of the image to be processed is an anomalous label, a third semantic matching score can be determined for the image to be processed and the reference image using the target image processing model. The third semantic matching score is determined based on the text features of the image to be processed and the visual features of the reference image, or vice versa. Then, when the third semantic matching score is less than the sixth threshold, the reference score difference is updated based on the first amplification factor to obtain the target score difference. The reference score difference is determined based on the difference (absolute value difference) between the image representation score of the image to be processed and the target score. The target score is obtained by updating the image representation score of the image to be processed based on the second amplification factor. Furthermore, the comparison result between the target score difference and the preset score difference is used to determine whether the two images are similar. Of course, if the third semantic matching score is greater than or equal to the sixth threshold, the comparison result between the difference (absolute value difference) between the target score and the image representation score of the reference image and the preset score difference is used to determine whether the two images are similar.

[0130] The business system outputs a similarity determination result and provides information related to the two compared images, such as their respective image representation scores, the difference between their scores (absolute difference), their respective semantic matching scores, their respective image category labels, and a third semantic matching score (if calculated). If the image to be processed is provided by the user, the user can determine whether there are any objections that need to be appealed based on reviewing the business system's output. If the image to be processed is provided by the business system, staff can determine whether there are any objections that need to be appealed based on reviewing the business system's output. The business system can then optimize the model based on the received appeal data.

[0131] As can be seen from the technical solutions provided in the embodiments of this application above, the target image processing model is a model with high generalization ability obtained through training. The target image processing model is used to process the image to be processed to determine the image representation score of the image to be processed. With the help of the adaptability and reliability of the target image processing model, the efficiency and effect of determining the image representation score of the image to be processed can be improved.

[0132] In one specific embodiment, the embodiments of this application can serve a product. This product provides an image similarity determination service based on an image processing model, adapting to multiple business scenarios such as content review, copyright protection, information filtering, and product monitoring. It provides users or business systems with an accurate, efficient, and low-cost image similarity determination solution, addressing the problems of high false positive rates, high operating costs, and poor adaptability of existing products.

[0133] The core functions of the product include: 1. Candidate Image Management Function: Supports batch entry, single upload, modification, deletion, query and expiration management of candidate images. During entry, preprocessing (text feature extraction, visual feature extraction, image representation score calculation and table storage) is automatically completed to ensure the timeliness and accuracy of candidate images. Users can classify and manage candidate images of different categories according to business needs to adapt to the judgment requirements of different scenarios.

[0134] 2. Image processing function: Supports users or business systems to provide images for processing in batches or individually, and automatically completes image preprocessing (size normalization, noise removal, blur repair) without manual intervention, thus improving processing efficiency; supports high-concurrency requests to meet the needs of large-scale image similarity judgment (specifically, it can support the judgment of 1000+ images per second).

[0135] 3. Similarity determination function: Based on the image processing model, it automatically completes multimodal feature extraction, deep fusion, score calculation and threshold discrimination of the image to be processed and the seed image, and outputs the determination result of "hit" or "miss" of the image. At the same time, it returns quantitative indicators such as the absolute value of the score difference, similarity ratio, and image-text semantic consistency score, so that users can understand the determination basis. It can also support users to view detailed determination process logs.

[0136] 4. Data statistics and visualization functions: Automatically calculates key indicators such as accuracy rate, false positive rate, number of hits, and amount of operating cost savings. It also adds special statistics for images with "high visual similarity and low text similarity" and displays them in chart form (such as line chart and bar chart) to help users intuitively understand the product application effect. It supports data export (formats include Excel and CSV) for business analysis and reporting.

[0137] 5. Model upgrade and configuration functions: Supports users to dynamically adjust threshold parameters and open the tag system according to business scenarios, and submit model upgrade requests (such as prompt text optimization, LLM fine-tuning). Model performance can be adapted without modifying the main structure of the model, thus improving the versatility of the product; configuration logs are provided to facilitate users to trace the history of parameter adjustments.

[0138] Product Application Flow: User uploads / Business system selects candidate image → Business system completes preprocessing of candidate image and stores related information in table → User uploads / Business system selects image to be processed → Business system completes preprocessing of image to be processed → Image processing model extracts image features and calculates image representation score → Business system calculates the difference (absolute value difference) between two image representation scores and compares it with the preset difference → Outputs similarity judgment result and quantitative index → ​​User views result / submits appeal → Business system collects appeal data for model optimization.

[0139] This application also provides an image processing model acquisition device, such as... Figure 8 As shown, the image processing model acquisition device 80 includes: The visual feature extraction module 801 is used to extract visual features of each sample image in the sample image set using a visual feature extraction network. The text feature extraction module 802 is used to extract text features from the sample image using a text feature extraction network; The feature processing module 803 is used to determine a first fusion feature and a first semantic matching score based on the visual features and the text features using a feature processing network; the first semantic matching score measures the degree of semantic matching between the visual features and the text features. Scoring module 804 is used to determine an image representation score based on the first fusion feature and the first semantic matching score using a scoring network; The parameter adjustment module 805 is used to adjust the parameters of the first model based on a first loss function and a second loss function until a target image processing model is obtained. The first model includes the visual feature extraction network, the text feature extraction network, the feature processing network, and the scoring network. The first loss function is constructed based on the difference between the first fused feature and the second fused feature of the reference image. The second loss function is constructed based on the difference between the first semantic matching score and the second semantic matching score of the reference image. The visual content similarity between the reference image and the sample image is greater than a first threshold, and the text content similarity is less than a second threshold.

[0140] In some embodiments, the visual features include global visual features and multiple first local visual features, the text features include global text features and multiple first local text features, and the feature processing module is further configured to: The global visual features and the global text features are fused using the feature processing network to obtain the global fused features; For each of the first local text features, the feature processing network is used to update the first local text features based on the text weights of the first local text features to obtain the corresponding second local text features. For each of the first local visual features, the feature processing network is used to obtain the corresponding second local visual feature based on the text weight of each of the first local text features and the attention weight between the first local visual feature and each of the first local text features. The feature processing network is used to fuse multiple second local visual features and multiple second local text features to obtain local fused features; The first fusion feature is obtained by using the feature processing network based on the global fusion feature and the local fusion feature.

[0141] In some embodiments, the text weight of the first local text feature is determined by the following steps: The text weight of the first local text feature is determined based on whether the local text is located within a visually salient region of the sample image; the local text is used to extract the first local text feature; or... The text weight of the first local text feature is determined based on the importance of the local text; or, The first weight of the first local text feature is determined based on whether the local text is located in the visually salient region of the sample image; the second weight of the first local text feature is determined based on the importance of the local text; and the text weight of the first local text feature is obtained based on the first weight and the second weight.

[0142] In some embodiments, the parameter adjustment module is further configured to: The parameters of the second model are adjusted based on the first loss function and the second loss function until the target image processing model is obtained; the target image processing model includes the visual feature extraction network and the parameter-adjusted second model, and the second model includes the text feature extraction network, the feature processing network and the scoring network.

[0143] In some embodiments, the device is further configured to: The feature processing network is used to determine image category labels based on the visual features and the text features; The parameter adjustment module is also used for: The parameters of the first model are adjusted based on the first loss function, the second loss function, and the third loss function until the target image processing model is obtained; the third loss function is constructed based on the difference between the image category label and the image category information carried by the sample image.

[0144] In some embodiments, the feature processing network is obtained through the following steps: For each first image in the candidate image set, the visual features and text features corresponding to the first image are fused using a preset network to obtain the fused features of the first image; A fourth loss function is constructed based on the difference between the fusion features of the first image and the fusion features of the second image; the visual content similarity between the second image and the first image is greater than the third threshold, and the text content similarity is less than the fourth threshold; The parameters of the preset network are adjusted based on the fourth loss function until the feature processing network is obtained.

[0145] In some embodiments, the device is further configured to: The first text is obtained by performing optical character recognition on the sample image; The first text is corrected based on the preset prompt text using a large language model to obtain the second text; The step of extracting text features from the sample image using a text feature extraction network includes: The text feature extraction network is used to extract features from the second text to obtain the text features.

[0146] It should be noted that the image processing model acquisition device provided in the above embodiments is only illustrated by the division of the above functional modules when performing the corresponding steps. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing model acquisition device and the image processing model acquisition method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0147] This application also provides an image processing apparatus, such as... Figure 9 As shown, the image processing apparatus 90 includes: Image acquisition module 901 is used to acquire the image to be processed; Image processing module 902 is used to determine the image representation score of the image to be processed using the target image processing model obtained by training according to the image processing model acquisition method described in steps 201-205 above.

[0148] In some embodiments, the device is further configured to: If the visual feature similarity between the image to be processed and the reference image is greater than a fifth threshold, the target image processing model is used to determine a third semantic matching score between the image to be processed and the reference image; the third semantic matching score is determined based on the text features of the image to be processed and the visual features of the reference image, or the visual features of the image to be processed and the text features of the reference image. If the third semantic matching score is less than the sixth threshold, the target score difference is obtained by updating the benchmark difference based on the first amplification factor; the benchmark difference is determined based on the difference between the image representation score of the image to be processed and the image representation score of the benchmark image. If the target difference is less than a preset difference, the image to be processed is determined to be a similar image to the reference image.

[0149] In some embodiments, the device is further configured to: If the image category label of the image to be processed is an anomalous label, the image representation score of the image to be processed is updated based on the second magnification factor to obtain the target score; If the difference between the target score and the image representation score of the reference image is less than a preset difference, the image to be processed is determined to be a similar image to the reference image.

[0150] It should be noted that the image processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when performing the corresponding steps. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing apparatus and image processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0151] Figure 10 A structural block diagram of an electronic device provided in an embodiment of this application is shown.

[0152] Typically, an electronic device 2000 includes a processor 2010 and a memory 2020.

[0153] Processor 2010 may include one or at least two processing cores, such as a quad-core processor or an octa-core processor. Processor 2010 may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 2010 may also include a main processor and coprocessors. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. Processor 2010 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content required for display on the screen. Processor 2010 may also include an artificial intelligence processor, which handles computational operations related to machine learning.

[0154] The memory 2020 may include at least one computer-readable storage medium. The computer-readable storage medium is capable of storing data and supports the reading of the stored data by a related processing module. The computer-readable storage medium can store data for a long period or a short period. The computer-readable storage medium can be a removable medium or a non-removable medium. The computer-readable storage medium can be a magnetic storage medium, an optical storage medium, a semiconductor storage medium, etc.

[0155] The electronic device 2000 may also include a power supply component 2030 (which is configured to perform power management of the electronic device 2000), a wired or wireless network interface 2040 (which is configured to connect the electronic device 2000 to a network), and an input / output (I / O) interface 2050.

[0156] In some embodiments, the memory 2020 stores a computer program that is loaded and executed by the processor 2010 to implement the methods described above.

[0157] This application also provides a computer-readable storage medium storing a computer program that is loaded and executed by a processor to implement the above-described method.

[0158] For example, computer storage media may be random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, high-density digital video discs (DVDs) or other optical storage, magnetic tape cassettes, magnetic tapes, disks, or other magnetic storage devices. Of course, those skilled in the art will understand that computer storage media are not limited to the above-mentioned types.

[0159] This application also provides a computer program product, which includes a computer program that is loaded and executed by a processor to implement the above-described method.

[0160] The computer program described above can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages. Programming languages ​​include object-oriented programming languages—such as Smalltalk, C+, etc.—and conventional procedural programming languages—such as the "C" language or similar programming languages. The computer program can execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can execute the computer program to implement various aspects of this application by utilizing the state information of the computer program.

[0161] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, and electronic devices according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by a computer program.

[0162] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for obtaining an image processing model, characterized in that, The method includes: For each sample image in the sample image set, a visual feature extraction network is used to extract the visual features of the sample image; Text features of the sample images are extracted using a text feature extraction network; A feature processing network is used to determine a first fusion feature and a first semantic matching score based on the visual features and the text features; the first semantic matching score measures the degree of semantic matching between the visual features and the text features. The scoring network is used to determine the image representation score based on the first fusion feature and the first semantic matching score; The parameters of the first model are adjusted based on the first loss function and the second loss function until the target image processing model is obtained; the first model includes the visual feature extraction network, the text feature extraction network, the feature processing network and the scoring network. The first loss function is constructed based on the difference between the first fused feature and the second fused feature of the reference image. The second loss function is constructed based on the difference between the first semantic matching score and the second semantic matching score of the reference image. The visual content similarity between the reference image and the sample image is greater than a first threshold and the text content similarity is less than a second threshold.

2. The method according to claim 1, characterized in that, The visual features include global visual features and multiple first local visual features; the text features include global text features and multiple first local text features; the step of using a feature processing network to determine a first fusion feature based on the visual features and the text features includes: The global visual features and the global text features are fused using the feature processing network to obtain the global fused features; For each of the first local text features, the feature processing network is used to update the first local text features based on the text weights of the first local text features to obtain the corresponding second local text features. For each of the first local visual features, the feature processing network is used to obtain the corresponding second local visual feature based on the text weight of each of the first local text features and the attention weight between the first local visual feature and each of the first local text features. The feature processing network is used to fuse multiple second local visual features and multiple second local text features to obtain local fused features; The first fusion feature is obtained by using the feature processing network based on the global fusion feature and the local fusion feature.

3. The method according to claim 2, characterized in that, The text weight of the first local text feature is determined through the following steps: The text weight of the first local text feature is determined based on whether the local text is located within a visually salient region of the sample image; the local text is used to extract the first local text feature; or... The text weight of the first local text feature is determined based on the importance of the local text; or, The first weight of the first local text feature is determined based on whether the local text is located in the visually salient region of the sample image; the second weight of the first local text feature is determined based on the importance of the local text; and the text weight of the first local text feature is obtained based on the first weight and the second weight.

4. The method according to any one of claims 1-3, characterized in that, The step of adjusting the parameters of the first model based on the first loss function and the second loss function until the target image processing model is obtained includes: The parameters of the second model are adjusted based on the first loss function and the second loss function until the target image processing model is obtained; the target image processing model includes the visual feature extraction network and the parameter-adjusted second model, and the second model includes the text feature extraction network, the feature processing network and the scoring network.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: The feature processing network is used to determine image category labels based on the visual features and the text features; The step of adjusting the parameters of the first model based on the first loss function and the second loss function until the target image processing model is obtained includes: The parameters of the first model are adjusted based on the first loss function, the second loss function, and the third loss function until the target image processing model is obtained; the third loss function is constructed based on the difference between the image category label and the image category information carried by the sample image.

6. The method according to any one of claims 1-3, characterized in that, The feature processing network is obtained through the following steps: For each first image in the candidate image set, the visual features and text features corresponding to the first image are fused using a preset network to obtain the fused features of the first image; A fourth loss function is constructed based on the difference between the fusion features of the first image and the fusion features of the second image; the visual content similarity between the second image and the first image is greater than the third threshold, and the text content similarity is less than the fourth threshold; The parameters of the preset network are adjusted based on the fourth loss function until the feature processing network is obtained.

7. The method according to claim 1, characterized in that, Before extracting text features from the sample image using a text feature extraction network, the method further includes: The first text is obtained by performing optical character recognition on the sample image; The first text is corrected based on the preset prompt text using a large language model to obtain the second text; The step of extracting text features from the sample image using a text feature extraction network includes: The text feature extraction network is used to extract features from the second text to obtain the text features.

8. An image processing method, characterized in that, The method includes: Obtain the image to be processed; The image representation score of the image to be processed is determined by training a target image processing model using the image processing model acquisition method according to any one of claims 1-7.

9. The method according to claim 8, characterized in that, The method further includes: If the visual feature similarity between the image to be processed and the reference image is greater than a fifth threshold, the target image processing model is used to determine a third semantic matching score between the image to be processed and the reference image; the third semantic matching score is determined based on the text features of the image to be processed and the visual features of the reference image, or the visual features of the image to be processed and the text features of the reference image. If the third semantic matching score is less than the sixth threshold, the target score difference is obtained by updating the benchmark difference based on the first amplification factor; the benchmark difference is determined based on the difference between the image representation score of the image to be processed and the image representation score of the benchmark image. If the target difference is less than a preset difference, the image to be processed is determined to be a similar image to the reference image.

10. The method according to claim 8, characterized in that, The method further includes: If the image category label of the image to be processed is an anomalous label, the image representation score of the image to be processed is updated based on the second magnification factor to obtain the target score; the image category label of the image to be processed is determined using the target image processing model obtained by training the image processing model acquisition method according to claim 5. If the difference between the target score and the image representation score of the reference image is less than a preset difference, the image to be processed is determined to be a similar image to the reference image.

11. An image processing model acquisition device, characterized in that, The device includes: The visual feature extraction module is used to extract the visual features of each sample image in the sample image set using a visual feature extraction network. The text feature extraction module is used to extract text features from the sample image using a text feature extraction network; The feature processing module is used to determine a first fusion feature and a first semantic matching score based on the visual features and the text features using a feature processing network; the first semantic matching score measures the degree of semantic matching between the visual features and the text features. The scoring module is used to determine the image representation score based on the first fusion feature and the first semantic matching score using a scoring network; The parameter adjustment module is used to adjust the parameters of the first model based on a first loss function and a second loss function until a target image processing model is obtained. The first model includes the visual feature extraction network, the text feature extraction network, the feature processing network, and the scoring network. The first loss function is constructed based on the difference between the first fused feature and the second fused feature of the reference image. The second loss function is constructed based on the difference between the first semantic matching score and the second semantic matching score of the reference image. The visual content similarity between the reference image and the sample image is greater than a first threshold, and the text content similarity is less than a second threshold.

12. An image processing apparatus, characterized in that, The device includes: The image acquisition module is used to acquire the image to be processed. An image processing module is used to determine the image representation score of the image to be processed using a target image processing model obtained by training according to the image processing model acquisition method according to any one of claims 1-7.

13. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, which is loaded and executed by the processor to implement the image processing model acquisition method as described in any one of claims 1 to 7, or the image processing method as described in any one of claims 8 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the image processing model acquisition method as described in any one of claims 1 to 7, or the image processing method as described in any one of claims 8 to 10.

15. A computer program product, characterized in that, The computer program product includes a computer program that is loaded and executed by a processor to implement the image processing model acquisition method as described in any one of claims 1 to 7, or the image processing method as described in any one of claims 8 to 10.

Images