A model training method, image matching method, device and storage medium

Abstract

The present application discloses a model training method based on artificial intelligence technology, including: obtaining a first image to be trained; obtaining a first region segmentation result based on the first image to be trained through a semantic segmentation model; obtaining a first sample image and a second sample image from the first image to be trained according to the first region segmentation result, wherein the area proportion of the region of interest included in the first sample image and the second sample image is greater than or equal to a proportion threshold; updating the model parameters of the image matching model to be trained according to the first sample image and the second sample image until the model training conditions are met to obtain an image matching model. The present application also provides a method, device and medium for image matching. The present application can more specifically select images and extract representative image blocks as sample images, which is conducive to training a more robust image matching model.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a model training method, an image matching method, a device, and a storage medium. Background Art

[0002] With the rapid development of Internet technology, it is becoming increasingly difficult for users to find the content they need in the vast ocean of interactive images. Therefore, using computers to extract the visual content of images and organize them, while improving the efficiency of image retrieval and recommendation, has become a research hotspot in the field of Computer Vision (CV).

[0003] Currently, image matching models are commonly used to determine the similarity between two images. Specifically, self-supervised learning can be used to train image matching models. During self-supervised learning, two image blocks are randomly selected from the same image as positive samples, and one image block is randomly selected from two different images as negative samples.

[0004] However, randomly selecting image patches from an image is often not very representative. For example, you might select a core region (i.e., the area containing text and images) while leaving a blank area. Or, you might select two blank areas. Therefore, using unrepresentative samples for training results in poor model performance. Summary of the Invention

[0005] The embodiments of the present application provide a model training method, an image matching method, an apparatus, and a storage medium. The present application divides an image into a region of interest and a background region. Thus, when randomly selecting image blocks from an image, the image can be more specifically selected, and representative image blocks can be extracted as sample images, thereby facilitating the training of a more robust image matching model.

[0006] In view of this, the present application provides a model training method, comprising:

[0007] Obtaining a first image to be trained;

[0008] Based on the first image to be trained, obtaining a first region segmentation result through a semantic segmentation model, wherein the first region segmentation result is used to determine a region of interest and a background region in the first image to be trained, where the region of interest includes at least one of a text region and a picture region;

[0009] Obtaining, based on the first region segmentation result, a first sample image and a second sample image derived from the first image to be trained, wherein the area ratio of the region of interest included in the first sample image is greater than or equal to a ratio threshold, and the area ratio of the region of interest included in the second sample image is greater than or equal to the ratio threshold;

[0010] The model parameters of the image matching model to be trained are updated according to the first sample image and the second sample image until the model training conditions are met, thereby obtaining an image matching model.

[0011] Another aspect of the present application provides an image matching method, comprising:

[0012] Acquire a first image to be matched;

[0013] Acquire a second image to be matched;

[0014] Based on the first image to be matched and the second image to be matched, a matching result is determined by an image matching model, wherein the image matching model is trained using the method of the above aspect.

[0015] Another aspect of the present application provides a model training device, comprising:

[0016] An acquisition module, configured to acquire a first image to be trained;

[0017] The acquisition module is further configured to obtain a first region segmentation result based on the first image to be trained using the semantic segmentation model, wherein the first region segmentation result is used to determine a region of interest and a background region in the first image to be trained, where the region of interest includes at least one of a text region and a picture region;

[0018] The acquisition module is further configured to acquire, based on the first region segmentation result, a first sample image and a second sample image derived from the first image to be trained, wherein the area ratio of the region of interest included in the first sample image is greater than or equal to a ratio threshold, and the area ratio of the region of interest included in the second sample image is greater than or equal to the ratio threshold;

[0019] The training module is used to update the model parameters of the image matching model to be trained according to the first sample image and the second sample image until the model training conditions are met to obtain the image matching model.

[0020] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0021] An acquisition module, specifically configured to randomly select a first image and a second image from a first image to be trained;

[0022] Determining the area ratio of the region of interest in the first image according to the first region segmentation result, and determining the area ratio of the region of interest in the second image;

[0023] If the area ratio of the region of interest in the first image is greater than or equal to the ratio threshold, the first image is used as the first sample image;

[0024] If the area ratio of the region of interest in the second image is greater than or equal to the ratio threshold, the second image is used as the second sample image.

[0025] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0026] An acquisition module, specifically configured to randomly select a first image to be processed and a second image to be processed from the first image to be trained;

[0027] Determining the area ratio of the region of interest in the first image to be processed according to the first region segmentation result, and determining the area ratio of the region of interest in the second image to be processed;

[0028] If the area ratio of the region of interest in the first image to be processed is greater than or equal to the ratio threshold, performing data augmentation processing on the first image to be processed to obtain a first sample image;

[0029] If the area ratio of the region of interest in the second image to be processed is greater than or equal to the ratio threshold, data augmentation processing is performed on the second image to be processed to obtain a second sample image.

[0030] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0031] The training module is specifically configured to obtain a first feature map based on the first sample image through an encoding network included in the image matching model to be trained;

[0032] Obtaining a first feature vector based on the first feature map through a projection network included in the image matching model to be trained;

[0033] Based on the first eigenvector, obtaining a target eigenvector through a prediction network included in the image matching model to be trained;

[0034] Based on the second sample image, obtaining a second feature map through the encoding network included in the target model;

[0035] Based on the second feature map, obtaining a second feature vector through a projection network included in the target model;

[0036] According to the target feature vector and the second feature vector, the model parameters of the image matching model to be trained are updated by using a first loss function.

[0037] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0038] The acquisition module is further used to acquire a second image to be trained;

[0039] The acquisition module is further used to obtain a second region segmentation result based on the second image to be trained through the semantic segmentation model, wherein the second region segmentation result is used to determine the region of interest and the background region in the second image to be trained;

[0040] The acquisition module is further configured to acquire, based on the second region segmentation result, a third sample image derived from the second image to be trained, wherein a proportion of the region of interest included in the third sample image is greater than or equal to a proportion threshold;

[0041] The training module is specifically used to update the model parameters of the image matching model to be trained according to the first sample image, the second sample image and the third sample image until the model training conditions are met to obtain the image matching model.

[0042] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0043] a training module, specifically configured to obtain, based on a first sample image and a second sample image, a feature vector corresponding to the first sample image and a feature vector corresponding to the second sample image through an image matching model to be trained, wherein the first sample image and the second sample image belong to a positive sample image pair, and the positive sample image pair corresponds to a positive sample label;

[0044] Based on the first sample image and the third sample image, obtaining a feature vector corresponding to the first sample image and a feature vector corresponding to the third sample image through the image matching model to be trained, the first sample image and the third sample image belonging to a negative sample image pair, and the negative sample image pair corresponding to a negative sample label;

[0045] determining a first characteristic distance according to a characteristic vector corresponding to the first sample image and a characteristic vector corresponding to the second sample image;

[0046] determining a second characteristic distance according to a characteristic vector corresponding to the first sample image and a characteristic vector corresponding to the third sample image;

[0047] A second loss function is used to update model parameters of the image matching model to be trained according to the first feature distance, the second feature distance, the positive sample label, and the negative sample label.

[0048] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0049] The training module is specifically configured to obtain, based on the first sample image, an embedding vector corresponding to the first sample image through the image matching model to be trained;

[0050] Based on the second sample image, obtaining an embedding vector corresponding to the second sample image through the image matching model to be trained, wherein the second sample image is a positive sample image;

[0051] Based on the third sample image, obtaining an embedding vector corresponding to the third sample image through the image matching model to be trained, wherein the third sample image is a negative sample image;

[0052] Determining a first embedding distance according to the embedding vector corresponding to the first sample image and the embedding vector corresponding to the second sample image;

[0053] determining a second embedding distance according to the embedding vector corresponding to the first sample image and the embedding vector corresponding to the third sample image;

[0054] According to the first embedding distance and the second embedding distance, a third loss function is used to update the model parameters of the image matching model to be trained.

[0055] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0056] A training module is specifically configured to obtain a predicted classification result based on a first sample image and a second sample image by using an image matching model to be trained, wherein the first sample image and the second sample image correspond to an annotated classification label;

[0057] According to the predicted classification results and the marked classification labels, the fourth loss function is used to update the model parameters of the image matching model to be trained.

[0058] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0059] The acquisition module is further used to acquire a training sample image, wherein the training sample image includes M pixels, each pixel corresponds to a category label, the category label is used to indicate whether the pixel belongs to the region of interest or the background region, and M is an integer greater than 1;

[0060] The acquisition module is also used to obtain the category prediction probability of each pixel point through the semantic segmentation model to be trained based on the training sample image;

[0061] The training module is also used to update the model parameters of the semantic segmentation model to be trained using the fifth loss function according to the category prediction probability of each pixel point and the category annotation label of each pixel point until the model training conditions are met to obtain the semantic segmentation model.

[0062] Another aspect of the present application provides an image matching device, comprising:

[0063] An acquisition module, configured to acquire a first image to be matched;

[0064] The acquisition module is further used to acquire a second image to be matched;

[0065] The determination module is used to determine the matching result based on the first image to be matched and the second image to be matched by using an image matching model, wherein the image matching model is trained using the method of the above aspect.

[0066] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0067] A determination module is specifically configured to obtain a feature vector corresponding to the first image to be matched through an image matching model based on the first image to be matched;

[0068] Based on the second image to be matched, obtaining a feature vector corresponding to the second image to be matched through an image matching model;

[0069] Determining a target feature distance based on a feature vector corresponding to the first image to be matched and a feature vector corresponding to the second image to be matched;

[0070] If the target feature distance is less than or equal to the distance threshold, the matching result is determined to be a successful match;

[0071] If the target feature distance is greater than the distance threshold, the matching result is determined to be a matching failure.

[0072] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0073] a determination module, specifically configured to obtain a similarity score based on the first image to be matched and the second image to be matched using an image matching model;

[0074] If the similarity score is greater than or equal to the similarity threshold, the matching result is determined to be a successful match;

[0075] If the similarity score is less than the similarity threshold, the matching result is determined to be a matching failure.

[0076] Another aspect of the present application provides a computer device, comprising: a memory, a processor, and a bus system;

[0077] Wherein, the memory is used to store programs;

[0078] The processor is used to execute the program in the memory, and the processor is used to perform the above-mentioned methods according to the instructions in the program code;

[0079] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0080] Another aspect of the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0081] Another aspect of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0082] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0083] The present application provides a method for model training, which first obtains a first image to be trained, and then obtains a first region segmentation result based on the first image to be trained through a semantic segmentation model. The first region segmentation result is used to determine the region of interest and the background region in the first image to be trained. Based on the first region segmentation result, a first sample image and a second sample image derived from the first image to be trained are obtained. The region ratio of the region of interest included in the first sample image is greater than or equal to the ratio threshold, and the region ratio of the region of interest included in the second sample image is greater than or equal to the ratio threshold. Finally, the model parameters of the image matching model to be trained can be updated according to the first sample image and the second sample image until the model training conditions are met to obtain an image matching model. In the above manner, the image is divided into the region of interest and the background region. Thus, when an image block is randomly framed from the image, the image can be framed more specifically, and representative image blocks can be taken out as sample images, which is conducive to training a more robust image matching model. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 This is a schematic diagram of the architecture of the image matching system in an embodiment of the present application;

[0085] Figure 2 A schematic diagram of the process of model training and image matching in an embodiment of the present application;

[0086] Figure 3 This is a flow chart of the model training method in the embodiment of the present application;

[0087] Figure 4This is a schematic diagram of model training based on a semantic segmentation model in an embodiment of the present application;

[0088] Figure 5 This is a schematic diagram of randomly selecting image blocks in an embodiment of the present application;

[0089] Figure 6 This is another schematic diagram of randomly selecting image blocks in an embodiment of the present application;

[0090] Figure 7 This is a structural diagram of an image matching model to be trained in an embodiment of the present application;

[0091] Figure 8 This is another schematic diagram of randomly selecting image blocks in an embodiment of the present application;

[0092] Figure 9 This is another structural diagram of the image matching model to be trained in an embodiment of the present application;

[0093] Figure 10 This is another structural diagram of the image matching model to be trained in an embodiment of the present application;

[0094] Figure 11 This is another structural diagram of the image matching model to be trained in an embodiment of the present application;

[0095] Figure 12 A schematic diagram showing the region of interest and the background region in an embodiment of the present application;

[0096] Figure 13 A schematic diagram of a flow chart of an image matching method in an embodiment of the present application;

[0097] Figure 14 A schematic diagram of the structure of the image matching model in the embodiment of the present application;

[0098] Figure 15 This is another structural diagram of the image matching model in the embodiment of the present application;

[0099] Figure 16 This is a schematic diagram of a model training device in an embodiment of the present application;

[0100] Figure 17 A schematic diagram of an image matching device in an embodiment of the present application;

[0101] Figure 18 A schematic diagram of the structure of the server in the embodiment of the present application;

[0102] Figure 19 This is a structural diagram of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0103] The embodiments of the present application provide a model training method, an image matching method, an apparatus, and a storage medium. The present application divides an image into a region of interest and a background region. Thus, when randomly selecting image blocks from an image, the image can be more specifically selected, and representative image blocks can be extracted as sample images, thereby facilitating the training of a more robust image matching model.

[0104] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "corresponding to" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0105] In recent years, the field of artificial intelligence (AI) has garnered unprecedented attention from all sectors of society, with machines continuously emulating and, to a certain extent, surpassing unique human strengths and skills. Computer vision (CV) technology is a core area of AI development. CV is the science of enabling machines to "see." More specifically, it involves using cameras and computers to replace the human eye in tasks such as object recognition, tracking, and measurement. Further image processing is performed to transform the computer-generated images into images more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, and smart transportation. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition. Image matching can be achieved with the help of CV technology. The following will introduce some application scenarios of the image matching method provided by this application.

[0106] Scenario 1: Image search application scenario;

[0107] (1) E-commerce shopping: Users can search for similar products and compare them. That is, the user uploads a picture of product A, and the server uses the image matching model to match product A with other product pictures stored in the backend, thereby finding product pictures with high similarity.

[0108] (2) Entertainment: Users can find the names of movies or TV series based on screenshots. That is, the user uploads screenshot A, and the server uses the image matching model to match screenshot A with other screenshots stored in the backend, thereby finding the screenshot with the highest similarity.

[0109] (3) Image retrieval: Upload image A to the image search website, and the server calls the image matching model to match image A with other images stored in the background, thereby finding images with higher similarity as the retrieval results.

[0110] Scenario 2: Application scenario of identifying graphic and text content;

[0111] Identifying Interesting Content: Capture images from web pages. Images include at least one of text and images. Text images refer to text presented in image form. The server uses an image matching model to perform pairwise matching of text images with stored text images, identifying images with the highest similarity.

[0112] In order to reduce the image mismatching during retrieval in the above scenario, it is necessary to train a more robust image matching model. This application proposes a model training method that is applied to Figure 1The image matching system shown in the figure includes a server and a terminal device, and the client is deployed on the terminal device, wherein the client can be run on the terminal device in the form of a browser, or can be run on the terminal device in the form of an independent application (APP), etc. The specific presentation form of the client is not limited here. The server involved in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal device can be a smart phone, tablet computer, laptop computer, PDA, personal computer, smart TV, smart watch, vehicle-mounted device, wearable device, etc., but is not limited to this. The terminal device and the server can be directly or indirectly connected by wired or wireless communication, and this application does not limit this. The number of servers and terminal devices is also not limited. The solution provided in this application can be completed independently by the terminal device, can be completed independently by the server, or can be completed by the terminal device and the server in cooperation. This application does not make any specific limitations on this.

[0113] Based on this, combined Figure 1 The image matching system shown in Figure 1 can realize model training and image matching. For easier understanding, please refer to Figure 2 , Figure 2 The following is a flow chart of model training and image matching in an embodiment of the present application, specifically:

[0114] In step S1, it is first necessary to collect training sample images in a target scene, wherein the target scene may refer to a web page or a public account, etc., and the training sample images represent images captured in the target scene.

[0115] In step S2, some training sample images are selected from the training sample images for manual labeling. For example, 10,000 training sample images are selected for labeling, that is, the text area, picture area and background area in the training sample images are labeled respectively.

[0116] In step S3, a semantic segmentation model is trained using the labeled images to segment the text area, picture area, and background area in the training image. The trained semantic segmentation model is used to segment all the training images to obtain the segmented areas of each training image.

[0117] In step S4, during random sampling, it is detected whether the sum of the proportions of the text area and the picture area of the sample image is greater than or equal to the proportion threshold. If it is greater than or equal to the proportion threshold, the proportion meets the requirements. Therefore, a sample image that meets the requirements is selected for training to obtain an image matching model.

[0118] In step S5, the trained image matching model is used to extract image feature values of the two images, and the similarity between the two images is measured according to the Euclidean distance between the two image feature values.

[0119] Combined with the above introduction, the following will introduce the model training method in this application. Please refer to Figure 3 , an embodiment of the model training method in the embodiment of the present application includes:

[0120] 110. Obtain a first image to be trained;

[0121] In one or more embodiments, the model training device may select an image to be trained from the set of images to be trained as the first image to be trained, wherein the images to be trained included in the set of images to be trained may be page screenshots, or application screenshots, etc., which are not limited here.

[0122] It should be noted that the model training device provided in this application can be deployed on a server, or can be deployed on a terminal device, or can be deployed on a system consisting of a server and a terminal device, which is not limited here.

[0123] 120. Obtain a first region segmentation result based on the first image to be trained using a semantic segmentation model, wherein the first region segmentation result is used to determine a region of interest and a background region in the first image to be trained, where the region of interest includes at least one of a text region and an image region;

[0124] In one or more embodiments, a model training device inputs a first image to be trained into a semantic segmentation model, which outputs a first region segmentation result. The first region segmentation result is used to segment the first image to be trained into a region of interest and a background region. It is understood that the region of interest includes at least one of a text region and an image region, and the text region includes text content presented in the form of an image.

[0125] 130. Obtain a first sample image and a second sample image from the first to-be-trained image based on the first region segmentation result, wherein the area ratio of the region of interest included in the first sample image is greater than or equal to a ratio threshold, and the area ratio of the region of interest included in the second sample image is greater than or equal to the ratio threshold;

[0126] In one or more embodiments, the model training device divides the first training image into a region of interest and a background region based on the first region segmentation result. When selecting the first sample image and the second sample image, the region of interest included in the first sample image must have a region ratio greater than or equal to a ratio threshold, and the region of interest included in the second sample image must also have a region ratio greater than or equal to the ratio threshold. There are multiple ways to extract the first sample image and the second sample image, each of which is described below.

[0127] For example, a first sample image that meets the requirements (i.e., the area proportion of the region of interest is greater than or equal to the proportion threshold) can be randomly intercepted from the first image to be trained, and another second sample image that meets the requirements can be randomly intercepted from the first image to be trained.

[0128] For example, a first sample image that meets the requirements (i.e., the area ratio of the region of interest is greater than or equal to the ratio threshold) can be randomly selected from the first image to be trained, and then data augmentation processing is performed on the first sample image to obtain a second sample image.

[0129] 140. Update the model parameters of the image matching model to be trained according to the first sample image and the second sample image until the model training conditions are met, thereby obtaining the image matching model.

[0130] In one or more embodiments, a model training device inputs a first sample image and a second sample image into an image matching model to be trained, and the image matching model to be trained outputs a predicted value. A loss function is used to estimate the degree of inconsistency between the predicted value and the true value of the image matching model to be trained. The model training process continuously iterates and calculates using a gradient descent optimization algorithm to reduce the loss function.

[0131] It is understood that the model training conditions include an "exhaustion criterion" and an "observation criterion." For example, taking the "exhaustion criterion" as an example, if the number of model iterations reaches an iteration threshold (e.g., 10,000), the model training conditions are met, and thus, the updated model parameters are used as the model parameters of the image matching model. For example, taking the "observation criterion" as an example, if the loss value has converged, the model training conditions are met, and thus, the updated model parameters are used as the model parameters of the image matching model.

[0132] Based on this, for ease of introduction, please refer to Figure 4 , Figure 4This is a schematic diagram of model training based on a semantic segmentation model in an embodiment of the present application. As shown in the figure, the first image to be trained is input into the semantic segmentation model, and the semantic segmentation model outputs a first region segmentation result. The first region segmentation result includes a circled region of interest, and the region of interest includes the image area indicated by A1 and the text area indicated by A2. Thus, two image sections can be randomly selected from the first image to be trained, namely the first sample image and the second sample image, wherein the first sample image and the second sample image belong to a positive sample image pair. The image matching model is trained based on the positive sample images until the model training conditions are met to obtain an image matching model.

[0133] The present application provides a model training method. Through the above method, the image is divided into a region of interest and a background area. Thus, when randomly selecting image blocks from the image, the image can be selected more specifically, and representative image blocks can be taken out as sample images, which is conducive to training a more robust image matching model.

[0134] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, obtaining a first sample image and a second sample image from the first image to be trained according to the first region segmentation result may specifically include:

[0135] Randomly select a first image and a second image from the first to-be-trained image;

[0136] Determining the area ratio of the region of interest in the first image according to the first region segmentation result, and determining the area ratio of the region of interest in the second image;

[0137] If the area ratio of the region of interest in the first image is greater than or equal to the ratio threshold, the first image is used as the first sample image;

[0138] If the area ratio of the region of interest in the second image is greater than or equal to the ratio threshold, the second image is used as the second sample image.

[0139] In one or more embodiments, a method for directly extracting positive sample image pairs is introduced. As can be seen from the aforementioned embodiments, a first sample image and a second sample image that meet the requirements can be directly extracted from the image to be trained (for example, the first image to be trained).

[0140] Specifically, for easier understanding, see Figure 5 , Figure 5This is a schematic diagram of randomly selecting image blocks in an embodiment of the present application. As shown in the figure, the first region segmentation result can be obtained after the first image to be trained passes through the semantic segmentation model. Therefore, two image blocks are randomly selected from the first image to be trained. The two image blocks are the first image indicated by B1 and the second image indicated by B2. Based on this, according to the first region segmentation result, the area ratio of the region of interest in the first image and the area ratio of the region of interest in the second image can be determined. Figure 5 For example, assuming that the area ratio of the ROI in the first image is 100%, the area ratio of the ROI in the second image is 80%, and the ratio threshold is 70%, it can be seen that the area ratio of the ROI in the first image is greater than or equal to the ratio threshold. Therefore, the first image is used as the first sample image. Similarly, the area ratio of the ROI in the second image is greater than or equal to the ratio threshold. Therefore, the second image is used as the second sample image.

[0141] Secondly, in an embodiment of the present application, a method for directly extracting positive sample image pairs is provided. Through the above method, sample images that meet the requirements can be randomly extracted directly from the images to be trained. Thus, these sample images can be used for subsequent model training, thereby increasing the feasibility and operability of the solution.

[0142] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, obtaining a first sample image and a second sample image from the first image to be trained according to the first region segmentation result may specifically include:

[0143] Randomly select a first image to be processed and a second image to be processed from the first image to be trained;

[0144] Determining the area ratio of the region of interest in the first image to be processed according to the first region segmentation result, and determining the area ratio of the region of interest in the second image to be processed;

[0145] If the area ratio of the region of interest in the first image to be processed is greater than or equal to the ratio threshold, performing data augmentation processing on the first image to be processed to obtain a first sample image;

[0146] If the area ratio of the region of interest in the second image to be processed is greater than or equal to the ratio threshold, data augmentation processing is performed on the second image to be processed to obtain a second sample image.

[0147] In one or more embodiments, a method for extracting positive sample image pairs based on data augmentation is described. As can be seen from the aforementioned embodiments, a first image to be processed and a second image to be processed that meet the requirements can be extracted from a training image (e.g., a first training image). Data augmentation is then performed on each of the training images to obtain a first sample image and a second sample image.

[0148] Specifically, for easier understanding, see Figure 6 , Figure 6 This is another schematic diagram of randomly selecting image blocks in an embodiment of the present application. As shown in the figure, the first region segmentation result can be obtained after the first image to be trained passes through the semantic segmentation model. Therefore, two image blocks are randomly selected from the first image to be trained, and the two image blocks are the first image to be processed indicated by C1, and the second image to be processed indicated by C2. Based on this, according to the first region segmentation result, the area ratio of the region of interest in the first image to be processed and the area ratio of the region of interest in the second image to be processed can be determined. Figure 6 For example, assuming that the area proportion of the region of interest in the first image to be processed is 100%, the area proportion of the region of interest in the second image to be processed is 80%, and assuming that the proportion threshold is 70%, it can be seen that the area proportion of the region of interest in the first image to be processed is greater than or equal to the proportion threshold. Therefore, data augmentation processing is performed on the first image to be processed to obtain a first sample image, and data augmentation processing is performed on the second image to be processed to obtain a second sample image.

[0149] It should be noted that data augmentation processing includes but is not limited to Gaussian blur, color conversion, grayscale conversion and rotation, etc. Figure 6 Rotation is used as a data augmentation method, however, this should not be construed as limiting the present application.

[0150] Secondly, in an embodiment of the present application, a method for extracting positive sample image pairs based on data augmentation is provided. Through the above method, a sample image that meets the requirements can be randomly extracted from the image to be trained, and then data augmentation is performed on the sample image to obtain another sample image. Thus, these sample images can be used for subsequent model training, thereby increasing the feasibility and operability of the solution.

[0151] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, updating the model parameters of the image matching model to be trained according to the first sample image and the second sample image may specifically include:

[0152] Based on the first sample image, obtaining a first feature map through an encoding network included in the image matching model to be trained;

[0153] Obtaining a first feature vector based on the first feature map through a projection network included in the image matching model to be trained;

[0154] Based on the first eigenvector, obtaining a target eigenvector through a prediction network included in the image matching model to be trained;

[0155] Based on the second sample image, obtaining a second feature map through the encoding network included in the target model;

[0156] Based on the second feature map, obtaining a second feature vector through a projection network included in the target model;

[0157] According to the target feature vector and the second feature vector, the model parameters of the image matching model to be trained are updated by using a first loss function.

[0158] In one or more embodiments, a method for training an image matching model based on a bootstrap your own latent (BYOL) framework is described. In the BYOL framework, for an input image (e.g., a first training image), two images (e.g., a first sample image and a second sample image) are obtained using two random image augmentation strategies.

[0159] Specifically, for easier understanding, see Figure 7 , Figure 7 This is a structural diagram of the image matching model to be trained in the embodiment of the present application. As shown in the figure, the BYOL framework includes an online network and a target network, wherein: Figure 7 The image matching model to be trained shown in FIG is an online network. A first sample image is input into the encoding network (i.e., encoding network 1) included in the image matching model to be trained, thereby outputting a first feature map. The first feature map is then input into the projection network (i.e., projection network 1) included in the image matching model to be trained, thereby outputting a first feature vector. The first feature vector is then input into the prediction network included in the image matching model to be trained, thereby outputting a target feature vector.

[0160] Similarly, the second sample image is input to the encoding network (i.e., encoding network 2) included in the image matching model to be trained, thereby outputting a second feature map. The second feature map is then input to the projection network (i.e., projection network 2) included in the image matching model to be trained, thereby outputting a second feature vector.

[0161] Based on this, the first loss function can be used to calculate the loss value between the target feature vector and the second feature vector to update the model parameters of the image matching model to be trained. The first loss function can be an L2 loss function, specifically:

[0162]

[0163] Where θ represents the model parameters of the online network (i.e., the image matching model to be trained). ξ represents the model parameters of the target network. θ represents the first eigenvector. θ (z θ ) represents the target feature vector. z′ ξ represents the second eigenvector.

[0164] It should be noted that the encoding network may be a Residual Network-50 (Resnet-50). The projection network may include fully connected layers (FC), batch normalization (BN) layers, and a rectified linear unit (ReLU).

[0165] After the model training is completed, the image matching model to be trained (ie, the online network) can be used as the image matching model for subsequent image matching.

[0166] Secondly, in an embodiment of the present application, a method for training an image matching model based on a BYOL framework is provided. Through the above method, relying on the interaction and mutual learning between the two networks, a higher classification accuracy can be achieved in the absence of negative sample pairs, thereby improving the robustness of the image matching model.

[0167] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, another optional embodiment provided by the embodiment of the present application may further include:

[0168] Acquire a second image to be trained;

[0169] Based on the second image to be trained, obtaining a second region segmentation result through the semantic segmentation model, wherein the second region segmentation result is used to determine a region of interest and a background region in the second image to be trained;

[0170] Acquire a third sample image derived from the second to-be-trained image according to the second region segmentation result, wherein a proportion of the region of interest included in the third sample image is greater than or equal to a proportion threshold;

[0171] Updating the model parameters of the image matching model to be trained according to the first sample image and the second sample image until the model training conditions are met to obtain the image matching model may specifically include:

[0172] According to the first sample image, the second sample image, and the third sample image, the model parameters of the image matching model to be trained are updated until the model training conditions are met, thereby obtaining the image matching model.

[0173] In one or more embodiments, a method of jointly training using positive and negative image sample pairs is introduced. As can be seen from the aforementioned embodiments, a first image to be processed and a second image to be processed that meet the requirements can be taken out from the image to be trained (for example, the first image to be trained), and the first image to be processed and the second image to be processed constitute a positive sample image pair. Similarly, a first image to be processed (or, the second image to be processed, or, the fourth sample image) and a third image to be processed that meet the requirements can be taken out from two images to be trained (for example, the first image to be trained and the second image to be trained), and the first image to be processed (or, the second image to be processed, or, the fourth sample image) and the third image to be processed constitute a negative sample image pair. It can be understood that the two sample images included in the negative sample image pair are derived from different images to be processed.

[0174] Specifically, for easier understanding, see Figure 8 , Figure 8 This is another schematic diagram of randomly selecting image blocks in an embodiment of the present application. As shown in the figure, the first region segmentation result can be obtained after the first image to be trained passes through the semantic segmentation model. Thus, two image blocks that meet the requirements are randomly selected from the first image to be trained. For example, the two image blocks are the first sample image indicated by D1, and the second sample image indicated by D2. The first sample image and the second sample image together constitute a positive sample image pair. Similarly, the second region segmentation result can be obtained after the second image to be trained passes through the semantic segmentation model. Thus, an image block that meets the requirements is randomly selected from the first image to be trained. For example, the image block is the third sample image indicated by D3. In addition, an image block that meets the requirements can also be extracted from an image to be trained. Taking the first image to be trained as an example, illustratively, the first sample image can be directly extracted. Thus, the first sample image and the third sample image together constitute a negative sample image pair. Exemplarily, the second sample image can be directly extracted. Thus, the second sample image and the third sample image together constitute a negative sample image pair. Exemplarily, the fourth sample image may be directly extracted (ie, a new sample image is extracted), whereby the fourth sample image and the third sample image together constitute a negative sample image pair.

[0175] It can be understood that a sample image that meets the requirements specifically refers to an image in which the proportion of the region of interest is greater than or equal to the proportion threshold.

[0176] Assume that the first and second sample images together constitute a positive sample image pair, and the first and third sample images together constitute a negative sample image pair. Based on this, the model parameters of the image matching model to be trained are updated by combining the positive and negative sample image pairs until the model training conditions are met, thereby obtaining an image matching model.

[0177] Secondly, in an embodiment of the present application, a method of jointly training using positive and negative image sample pairs is provided. Through the above method, positive sample image pairs can be extracted from the same image to be trained, and negative sample image pairs can be extracted from different images to be trained respectively. Based on this, using positive sample image pairs and negative sample image pairs for joint training can improve the robustness of the image matching model.

[0178] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, updating the model parameters of the image matching model to be trained based on the first sample image, the second sample image, and the third sample image may specifically include:

[0179] Based on the first sample image and the second sample image, obtaining a feature vector corresponding to the first sample image and a feature vector corresponding to the second sample image through the image matching model to be trained, the first sample image and the second sample image belonging to a positive sample image pair, and the positive sample image pair corresponding to a positive sample label;

[0180] Based on the first sample image and the third sample image, obtaining a feature vector corresponding to the first sample image and a feature vector corresponding to the third sample image through the image matching model to be trained, the first sample image and the third sample image belonging to a negative sample image pair, and the negative sample image pair corresponding to a negative sample label;

[0181] determining a first characteristic distance according to a characteristic vector corresponding to the first sample image and a characteristic vector corresponding to the second sample image;

[0182] determining a second characteristic distance according to a characteristic vector corresponding to the first sample image and a characteristic vector corresponding to the third sample image;

[0183] A second loss function is used to update model parameters of the image matching model to be trained according to the first feature distance, the second feature distance, the positive sample label, and the negative sample label.

[0184] In one or more embodiments, a method for training an image matching model using a contrastive loss function in combination with positive and negative image sample pairs is described. As can be seen from the aforementioned embodiments, during self-supervised network training, for the image to be trained, a first sample image, a second sample image, and a third sample image that meet the requirements can be randomly selected. It will be understood that this application uses the example of the first and second sample images belonging to a positive sample image pair, and the first and third sample images belonging to a negative sample image pair.

[0185] Specifically, for easier understanding, see Figure 9 , Figure 9 This is another structural diagram of the image matching model to be trained in an embodiment of the present application. As shown in the figure, the image matching model to be trained includes sub-model 1 and sub-model 2. Among them, sample image 1 and sample image 2 are a sample image pair (for example, a positive sample image pair consisting of a first sample image and a second sample image, or a negative sample image pair consisting of a first sample image and a third sample image). Thus, sample image 1 is input into sub-model 1, and sub-model 1 outputs feature vector 1. Sample image 2 is input into sub-model 2, and sub-model 2 outputs feature vector 2. Based on feature vector 1 and feature vector 2, the Euclidean distance between the two sample images can be calculated. In practical applications, N groups of sample image pairs can be input into the image matching model to be trained respectively, thereby calculating the Euclidean distance between the sample image pairs based on the predicted feature vectors, and then updating the model parameters.

[0186] Based on this, the second loss function can be used to calculate the loss value between the Euclidean distance and the true label to update the model parameters of the image matching model to be trained. The second loss function can use the contrast loss function, specifically:

[0187]

[0188] Among them, L C represents the second loss function. y represents the sample pair label, for example, the positive sample label is 1 and the negative sample label is 0. d represents the Euclidean distance between two feature vectors. max(·) represents the maximum value.

[0189] It's understandable that the contrastive loss function is primarily used in feature extraction applications. That is, if similar sample images are similar in feature space after feature extraction, then the two sample images will remain similar; and if dissimilar sample images are dissimilar in feature space after feature extraction, then the two sample images will remain dissimilar. Similarly, this loss function can also well express the degree of match between paired sample images.

[0190] It should be noted that the image matching model to be trained belongs to a siamese neural network (SNN). Therefore, the image matching model to be trained can include two branches, that is, two sub-models. The network structure of the two sub-models is the same, and the model parameters are shared. The twin neural network needs to input two sample images as a sample image pair each time. The input sample image is mapped into a feature vector, and the sample image pair between the two feature vectors is used to represent the semantic difference between the input images. The twin neural network can be used to determine the similarity between the two sample images.

[0191] After the model training is completed, any sub-model can be extracted from the image matching model to be trained as the image matching model for subsequent image matching.

[0192] Again, in an embodiment of the present application, a method is provided for training an image matching model by combining positive and negative image sample pairs with a contrast loss function. Through the above method, the network structure of the twin neural network is used as the network structure of the image matching model. Therefore, the contrast loss function can be used to effectively process the relationship between paired sample images in the twin neural network, and to express the matching degree of paired sample images well, thereby being well used for training a model for extracting features.

[0193] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, updating the model parameters of the image matching model to be trained based on the first sample image, the second sample image, and the third sample image may specifically include:

[0194] Based on the first sample image, obtaining an embedding vector corresponding to the first sample image through the image matching model to be trained;

[0195] Based on the second sample image, obtaining an embedding vector corresponding to the second sample image through the image matching model to be trained, wherein the second sample image is a positive sample image;

[0196] Based on the third sample image, obtaining an embedding vector corresponding to the third sample image through the image matching model to be trained, wherein the third sample image is a negative sample image;

[0197] Determining a first embedding distance according to the embedding vector corresponding to the first sample image and the embedding vector corresponding to the second sample image;

[0198] determining a second embedding distance according to the embedding vector corresponding to the first sample image and the embedding vector corresponding to the third sample image;

[0199] According to the first embedding distance and the second embedding distance, a third loss function is used to update the model parameters of the image matching model to be trained.

[0200] In one or more embodiments, a method for training a twin network by combining positive and negative sample images and using a triple loss function is introduced. As can be seen from the aforementioned embodiments, in the process of self-supervised network training, for the image to be trained, a first sample image, a second sample image, and a third sample image that meet the requirements can be randomly selected. It is understandable that this application uses the first sample image as the reference image (anchor), the second sample image as the positive sample image, and the third sample image as the negative sample image as an example for explanation.

[0201] Specifically, for easier understanding, see Figure 10 , Figure 10 This is another structural diagram of the image matching model to be trained in an embodiment of the present application. As shown in the figure, the image matching model to be trained includes sub-model 1, sub-model 2 and sub-model 3. Among them, sample image 2 is the first sample image (i.e., the reference image), sample image 1 is the second sample image (i.e., the positive sample image), and sample image 3 is the third sample image (i.e., the negative sample image). Thus, sample image 1 is input into sub-model 1, and sub-model 1 outputs embedding vector 1. Sample image 2 is input into sub-model 2, and sub-model 2 outputs embedding vector 2. Sample image 3 is input into sub-model 3, and sub-model 3 outputs embedding vector 3. Based on embedding vector 1 and embedding vector 2, the embedding distance between the two sample images can be calculated. Based on embedding vector 2 and embedding vector 3, the embedding distance between the two sample images can also be calculated. In practical applications, N triple sample images can be input into the image matching model to be trained respectively, and the Euclidean distance between the triple sample images is calculated based on the predicted embedding vector, and then the model parameters are updated.

[0202] To improve the robustness of the model, it is necessary not only to distinguish between positive and negative sample images, but also to make the intra-class distribution more compact and the inter-class distribution more distant. Therefore, the image matching model to be trained can be changed to three inputs, using one positive sample image and two negative sample images during training, or using one negative sample image and two positive sample images. This application uses the example of using one negative sample image and two positive sample images, but this should not be understood as a limitation of this application.

[0203] Based on this, the third loss function can be used to calculate the loss value between the embedding distances and update the model parameters of the image matching model to be trained. The third loss function can use the triplet loss function, specifically:

[0204] L T=max(0,D(A,P)-D(A,N)+margin);

[0205]

[0206] Among them, L T Represents the third loss function. A represents a reference image (e.g., a first sample image). P represents a positive sample image (e.g., a second sample image). N represents a positive sample image (e.g., a second sample image). Margin represents a preset distance, for example, 1. D(A,P) represents the embedding distance between the reference image and the positive sample image. D(A,N) represents the embedding distance between the reference image and the negative sample image. Max(·) represents the maximum value. Net(A) represents the embedding vector of the reference image output by one sub-model. Net(P) represents the embedding vector of the positive sample image output by another sub-model. Net(N) represents the embedding vector of the negative sample image output by another sub-model.

[0207] It should be noted that the image matching model to be trained belongs to a twin neural network. Therefore, it can include three branches, that is, three sub-models. The two tri-networks have the same network structure and share model parameters. The twin neural network requires three sample images as input as a triplet of sample images.

[0208] After the model training is completed, any sub-model can be extracted from the image matching model to be trained as the image matching model for subsequent image matching.

[0209] Again, in an embodiment of the present application, a method is provided for training a twin network by combining positive and negative sample images and using a triple loss function. Through the above method, the network structure of the twin neural network is used as the network structure of the image matching model. Therefore, the triple loss function can be used to effectively process the relationship between the three associated sample images in the twin neural network, and to express the matching degree of the three associated sample images well, thereby being well used for training a model for extracting features.

[0210] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, updating the model parameters of the image matching model to be trained according to the first sample image and the second sample image may specifically include:

[0211] Based on the first sample image and the second sample image, obtaining a predicted classification result through the image matching model to be trained, the first sample image and the second sample image corresponding to the annotated classification label;

[0212] According to the predicted classification results and the marked classification labels, the fourth loss function is used to update the model parameters of the image matching model to be trained.

[0213] In one or more embodiments, a method for training a Siamese network using a loss function in conjunction with positive samples is described. As can be seen from the aforementioned embodiments, during network training, a first sample image and a second sample image that meet the requirements can be randomly selected from the training images. It will be appreciated that this application uses the example of a positive sample image pair in which the first and second sample images are present.

[0214] Specifically, for easier understanding, see Figure 11 , Figure 11 This is another structural diagram of the image matching model to be trained in an embodiment of the present application. As shown in the figure, the image matching model to be trained includes sub-model 1 and sub-model 2. Among them, sample image 1 and sample image 2 are a sample image pair (for example, a positive sample image pair consisting of a first sample image and a second sample image, or a negative sample image pair consisting of a first sample image and a third sample image). Thus, sample image 1 is input into sub-model 1, and sub-model 1 outputs feature vector 1. Sample image 2 is input into sub-model 2, and sub-model 2 outputs feature vector 2. Feature vector 1 and feature vector 2 are input into the classification network together, and the classification network outputs the predicted classification result. Among them, the classification network includes a fully connected layer. The predicted classification result can be expressed as a similarity score greater than or equal to 0 and less than or equal to 1. The larger the similarity score, the higher the degree of similarity.

[0215] In practical applications, N groups of sample image pairs can be input into the image matching model to be trained respectively, thereby updating the model parameters according to the distance between the predicted classification result and the annotated classification label.

[0216] Based on this, the fourth loss function can be used to calculate the loss value between the Euclidean distance and the true label to update the model parameters of the image matching model to be trained. In one case, the fourth loss function can use the mean squared error (MSE) loss function, specifically:

[0217]

[0218] Among them, L M Represents the fourth loss function. N represents the number of sample image pairs. i represents the i-th sample image pair. Represents the labeled classification label of the i-th sample image pair, the labeled classification label of the positive sample image pair is 1, and the labeled classification label of the negative sample image pair is 0. iRepresents the predicted classification result of the i-th sample image pair, and the predicted classification result is a similarity score greater than or equal to 1 and less than or equal to 1.

[0219] In another case, the fourth loss function may adopt the mean absolute error (MAE) loss function, specifically:

[0220]

[0221] Among them, L M Represents the fourth loss function. N represents the number of sample image pairs. i represents the i-th sample image pair. Represents the labeled classification label of the i-th sample image pair, the labeled classification label of the positive sample image pair is 1, and the labeled classification label of the negative sample image pair is 0. i Represents the predicted classification result of the i-th sample image pair, and the predicted classification result is a similarity score greater than or equal to 1 and less than or equal to 1.

[0222] It is understandable that the fourth loss function may also be other types of loss functions, which are not limited here.

[0223] It should be noted that the image matching model to be trained belongs to a twin neural network. Therefore, the image matching model to be trained can include two branches and a classification network, that is, two sub-models and a classification network. The network structure of the two sub-models is the same, and the model parameters are shared. The twin neural network needs to input two sample images as a sample image pair each time. After mapping the input sample image to a feature vector, it is input into the classification network respectively, and the classification network outputs the predicted classification result. The twin neural network can be used to determine the similarity between two sample images.

[0224] After the model training is completed, the image matching model to be trained can be used as the image matching model for subsequent image matching.

[0225] Secondly, in an embodiment of the present application, a method for training a twin network by combining positive samples and using a loss function is provided. Through the above method, the network structure of the twin neural network is used as the network structure of the image matching model. Therefore, the cross-entropy loss function can be used to effectively process the relationship between paired sample images in the twin neural network, and express the matching degree of paired sample images well, so that it can be used well for training a model for outputting image similarity scores.

[0226] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, another optional embodiment provided by the embodiment of the present application may further include:

[0227] Obtain a training sample image, wherein the training sample image includes M pixels, each pixel corresponds to a category label, the category label is used to indicate whether the pixel belongs to the region of interest or the background region, and M is an integer greater than 1;

[0228] Based on the training sample image, the category prediction probability of each pixel is obtained through the semantic segmentation model to be trained;

[0229] According to the category prediction probability of each pixel and the category annotation label of each pixel, the fifth loss function is used to update the model parameters of the semantic segmentation model to be trained until the model training conditions are met to obtain the semantic segmentation model.

[0230] In one or more embodiments, a method for training a semantic segmentation model is introduced. As can be seen from the above embodiments, the semantic segmentation model can be used to divide an image into a region of interest and a background region. The following describes how to train the semantic segmentation model.

[0231] Specifically, we first need to collect a certain number (for example, greater than or equal to 50,000) of training sample images. The training sample images should be selected as randomly as possible to cover various cases in the target scene. In order to better select representative image blocks in the process of self-supervised learning similarity, we need to divide and annotate the areas of the training sample images in advance during the training process to obtain the regions of interest or background areas. Furthermore, if the regions of interest include text areas and image areas, we need to annotate three types of areas, namely text areas, image areas, and background areas. For ease of understanding, please refer to Figure 12 , Figure 12 This is a schematic diagram of marking the region of interest and background area in an embodiment of the present application. As shown in the figure, the area indicated by E1 is the picture area, the area indicated by E2 is the text area, and the remaining part is the background area.

[0232] It should be noted that labeling all training sample images often requires a lot of manpower. Therefore, you can consider labeling only some training sample images (for example, 10,000 images), use these labeled data to train a semantic segmentation model to learn the text area, picture area and background area, and then use the semantic segmentation model to segment the image.

[0233] The labeled training sample images are used as the input of the semantic segmentation model to be trained, and the labeling results are used as supervision information. The category label of each pixel in the text area can be set to "1", the category label of each pixel in the image area can be set to "2", and the category label of each pixel in the background area can be set to "0". Based on this, the fifth loss function can be used to calculate the loss value between the category prediction probability of each pixel and the category label, and the model parameters of the semantic segmentation model to be trained can be updated. The fifth loss function can adopt the cross entropy loss function, which is specifically:

[0234]

[0235] Among them, L E represents the fifth loss function. I represents the training sample image. S represents the labeled region (e.g., text region, image region, and background region). C represents the total number of region categories (e.g., three categories). c represents the cth region category. f u,c (I) represents the probability that the label y of pixel u in the image described by the semantic segmentation model to be trained belongs to the cth region category, that is, f u,c (I) = p(y u =c\I). S C Represents the set of pixels labeled as the c-th region category.

[0236] It should be noted that the semantic segmentation model provided in this application includes but is not limited to a fully convolution network (FCN), a segmentation network (SegNet), Deeplab, a pyramid scene parsing network (PSPNet), and a high-resource network (HRNet), etc., and is not limited here.

[0237] Secondly, in an embodiment of the present application, a method for training a semantic segmentation model is provided. Through the above method, it can be trained to not only segment the region of interest and the background area from the image, but also segment the text area and picture area in the region of interest, so that the randomly sampled image blocks can cover more effective content. As a result, the image can be framed more specifically, and representative image blocks can be taken out as sample images, which is conducive to training a more robust image matching model.

[0238] In combination with the above introduction, the image matching method in this application will be introduced below. Figure 13, an embodiment of the image matching method in the embodiment of the present application includes:

[0239] 210. Obtain a first image to be matched;

[0240] In one or more embodiments, the image matching device obtains a first image to be matched, wherein the first image to be matched may be an image uploaded by a user, or the first image to be matched may be an image captured from a web page, or the first image to be matched may be an image obtained through other means, which is not limited here.

[0241] It should be noted that the image matching device provided in this application can be deployed on a server, or can be deployed on a terminal device, or can be deployed on a system consisting of a server and a terminal device, which is not limited here.

[0242] 220. Obtain a second image to be matched;

[0243] In one or more embodiments, the image matching device obtains a second image to be matched, wherein the second matching image may be an image uploaded by a user, or the second image to be matched may be an image stored in the background, or the second image to be matched may be an image obtained through other means, which is not limited here.

[0244] 230. Based on the first image to be matched and the second image to be matched, determine a matching result using an image matching model, wherein the image matching model is trained using the method provided in the above embodiment.

[0245] In one or more embodiments, the image matching device uses the first image to be matched and the second image to be matched as inputs to an image matching model. In one embodiment, the image matching model can support two inputs simultaneously, that is, the first image to be matched and the second image to be matched are input to the image matching model together. In another embodiment, the image matching model can support one input, that is, the first image to be matched is input to the image matching model first, and then the second image to be matched is input to the image matching model. Based on the results output by the image matching model, matching results corresponding to the two images to be matched are generated.

[0246] In image retrieval scenarios, if a user inputs an image (i.e., the first image to be matched), similar images can be retrieved. In image classification scenarios, image matching models can be used to cluster images with high similarity, thereby improving the efficiency and accuracy of image classification.

[0247] In an embodiment of the present application, a method for image matching is provided. By dividing an image into a region of interest and a background region, the randomly selected image blocks can be more specifically selected, and representative image blocks can be extracted as sample images, thereby facilitating the training of a more robust image matching model. This reduces the occurrence of mismatches during model inference.

[0248] Optionally, in the above Figure 13 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, determining a matching result by an image matching model based on the first image to be matched and the second image to be matched may specifically include:

[0249] Based on the first image to be matched, obtaining a feature vector corresponding to the first image to be matched through an image matching model;

[0250] Based on the second image to be matched, obtaining a feature vector corresponding to the second image to be matched through an image matching model;

[0251] Determining a target feature distance based on a feature vector corresponding to the first image to be matched and a feature vector corresponding to the second image to be matched;

[0252] If the target feature distance is less than or equal to the distance threshold, the matching result is determined to be a successful match;

[0253] If the target feature distance is greater than the distance threshold, the matching result is determined to be a matching failure.

[0254] In one or more embodiments, a method for determining a matching result based on the distance between features is introduced. As can be seen from the aforementioned embodiments, the image matching model can be a sub-model in the image matching model to be trained. Based on this, the first image to be matched and the second image to be matched need to be input into the image matching model respectively, thereby outputting two feature vectors. Therefore, the target feature distance between the two feature vectors can be calculated using the calculation formula of the Euclidean distance. It can be understood that the target feature distance can be the Euclidean distance, which is also called the Euclidean distance. It is a common distance metric used to measure the absolute distance between two objects in a multidimensional space. The smaller the target feature distance, the greater the similarity between the two images to be matched. Conversely, the larger the target feature distance, the smaller the similarity between the two images to be matched.

[0255] Specifically, for easier understanding, see Figure 14 , Figure 14This is a schematic diagram of the structure of the image matching model in an embodiment of the present application. As shown in the figure, a first image to be matched is input into a trained image matching model, and the image matching model outputs a feature vector corresponding to the first image to be matched. Similarly, a second image to be matched is input into the trained image matching model, and the image matching model outputs a feature vector corresponding to the second image to be matched. Based on this, the Euclidean distance formula can be used to calculate the target feature distance between the feature vector corresponding to the first image to be matched and the feature vector corresponding to the second image to be matched.

[0256] If the target feature distance is less than or equal to the distance threshold, the match result is determined to be a successful match. Conversely, if the target feature distance is greater than the distance threshold, the match result is determined to be a failed match. Since the smaller the target feature distance, the more similar the images are, the target feature distance can be used to determine the image to be matched that is closest to the image to be matched.

[0257] Secondly, embodiments of the present application provide a method for determining matching results based on inter-feature distance. In this method, the image matching model directly outputs feature vectors of the two images to be matched, and the inter-feature distance is then determined based on the feature vectors of the two images to be matched. Because a smaller inter-feature distance indicates a higher degree of similarity, the image matching model can output a feature vector to determine the matching result of the two images to be matched, thereby improving the accuracy of image matching.

[0258] Optionally, in the above Figure 13 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, determining a matching result by an image matching model based on the first image to be matched and the second image to be matched may specifically include:

[0259] Based on the first image to be matched and the second image to be matched, obtaining a similarity score through an image matching model;

[0260] If the similarity score is greater than or equal to the similarity threshold, the matching result is determined to be a successful match;

[0261] If the similarity score is less than the similarity threshold, the matching result is determined to be a matching failure.

[0262] In one or more embodiments, a method for determining a matching result based on a similarity score is described. As can be seen from the aforementioned embodiments, the image matching model can be a trained image matching model, i.e., one comprising two sub-models. Based on this, a first image to be matched and a second image to be matched are input to the image matching model, thereby outputting a similarity score. Therefore, the similarity score between the two images to be matched can be calculated using the cosine similarity calculation formula. It is understood that the similarity score can be cosine similarity. Cosine similarity first requires vectorizing the images to be matched and then evaluating their similarity by calculating the cosine of the angle between the two feature vectors. Cosine similarity is typically used in positive space. Therefore, under normal circumstances, the cosine similarity value ranges from 0 to 1, where the cosine value of an angle of 0 degrees is 1, and the cosine value of any other angle is no greater than 1. That is, the greater the similarity score, the greater the similarity between the two images to be matched; conversely, the smaller the similarity score, the smaller the similarity between the two images to be matched.

[0263] Specifically, for easier understanding, see Figure 15 , Figure 15 This is another structural schematic diagram of the image matching model in an embodiment of the present application. As shown in the figure, the first image to be matched and the second image to be matched are input into the trained image matching model, and the image matching model outputs the similarity score between the first image to be matched and the second image to be matched.

[0264] If the similarity score is greater than or equal to the similarity threshold, the match result is determined to be a successful match. Conversely, if the similarity score is less than the similarity threshold, the match result is determined to be a failed match. Since the larger the similarity score, the more similar the images are, the similarity score can be used to determine another image to be matched that is closest to the image to be matched.

[0265] Secondly, in an embodiment of the present application, a method for determining the matching result based on the similarity score is provided. Through the above method, the image matching model directly outputs the similarity score between the two images to be matched. Since the larger the similarity score, the higher the degree of similarity, the image matching model can directly output the similarity score to determine the matching result of the two images to be matched, thereby improving the accuracy of image matching.

[0266] The model training device in this application is described in detail below. Figure 16 , Figure 16 This is a schematic diagram of an embodiment of a model training device in an embodiment of the present application. The model training device 30 includes:

[0267] An acquisition module 310 is configured to acquire a first image to be trained;

[0268] The acquisition module 310 is further configured to obtain a first region segmentation result based on the first image to be trained using the semantic segmentation model, wherein the first region segmentation result is used to determine a region of interest and a background region in the first image to be trained, where the region of interest includes at least one of a text region and a picture region;

[0269] The acquisition module 310 is further configured to acquire, based on the first region segmentation result, a first sample image and a second sample image derived from the first image to be trained, wherein the area ratio of the region of interest included in the first sample image is greater than or equal to a ratio threshold, and the area ratio of the region of interest included in the second sample image is greater than or equal to the ratio threshold;

[0270] The training module 320 is used to update the model parameters of the image matching model to be trained according to the first sample image and the second sample image until the model training conditions are met to obtain the image matching model.

[0271] The present application provides a model training device, which uses the above-mentioned device to divide an image into a region of interest and a background region. Therefore, when randomly selecting image blocks from an image, the image can be selected more specifically, and representative image blocks can be taken out as sample images, which is conducive to training a more robust image matching model.

[0272] Optionally, in the above Figure 16 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0273] An acquisition module 310 is specifically configured to randomly select a first image and a second image from a first image to be trained;

[0274] Determining the area ratio of the region of interest in the first image according to the first region segmentation result, and determining the area ratio of the region of interest in the second image;

[0275] If the area ratio of the region of interest in the first image is greater than or equal to the ratio threshold, the first image is used as the first sample image;

[0276] If the area ratio of the region of interest in the second image is greater than or equal to the ratio threshold, the second image is used as the second sample image.

[0277] The present application provides a model training device. By using the above device, sample images that meet the requirements can be randomly extracted directly from the images to be trained. Thus, these sample images can be used for subsequent model training, thereby increasing the feasibility and operability of the solution.

[0278] Optionally, in the above Figure 16On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0279] An acquisition module 310 is specifically configured to randomly select a first image to be processed and a second image to be processed from the first image to be trained;

[0280] Determining the area ratio of the region of interest in the first image to be processed according to the first region segmentation result, and determining the area ratio of the region of interest in the second image to be processed;

[0281] If the area ratio of the region of interest in the first image to be processed is greater than or equal to the ratio threshold, performing data augmentation processing on the first image to be processed to obtain a first sample image;

[0282] If the area ratio of the region of interest in the second image to be processed is greater than or equal to the ratio threshold, data augmentation processing is performed on the second image to be processed to obtain a second sample image.

[0283] The present application provides a model training device. Using the above device, a sample image that meets the requirements can be randomly extracted from the image to be trained, and then data augmentation is performed on the sample image to obtain another sample image. Thus, these sample images can be used for subsequent model training, thereby increasing the feasibility and operability of the solution.

[0284] Optionally, in the above Figure 16 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0285] The training module 320 is specifically configured to obtain a first feature map based on the first sample image through an encoding network included in the image matching model to be trained;

[0286] Obtaining a first feature vector based on the first feature map through a projection network included in the image matching model to be trained;

[0287] Based on the first eigenvector, obtaining a target eigenvector through a prediction network included in the image matching model to be trained;

[0288] Based on the second sample image, obtaining a second feature map through the encoding network included in the target model;

[0289] Based on the second feature map, obtaining a second feature vector through a projection network included in the target model;

[0290] According to the target feature vector and the second feature vector, the model parameters of the image matching model to be trained are updated by using a first loss function.

[0291] The present application provides a model training device. By using the above-mentioned device, relying on the interaction and mutual learning between the two networks, a higher classification accuracy can be achieved in the absence of negative sample pairs, thereby improving the robustness of the image matching model.

[0292] Optionally, in the above Figure 16 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0293] The acquisition module 310 is further configured to acquire a second image to be trained;

[0294] The acquisition module 310 is further configured to acquire a second region segmentation result based on the second image to be trained using the semantic segmentation model, wherein the second region segmentation result is used to determine a region of interest and a background region in the second image to be trained;

[0295] The acquisition module 310 is further configured to acquire a third sample image derived from the second image to be trained based on the second region segmentation result, wherein a proportion of the region of interest included in the third sample image is greater than or equal to a proportion threshold;

[0296] The training module 320 is specifically configured to update the model parameters of the image matching model to be trained according to the first sample image, the second sample image, and the third sample image until the model training conditions are met, thereby obtaining the image matching model.

[0297] The present application provides a model training device. Using the above device, positive sample image pairs can be extracted from the same image to be trained, and negative sample image pairs can be extracted from different images to be trained. Based on this, positive sample image pairs and negative sample image pairs are used for joint training to improve the robustness of the image matching model.

[0298] Optionally, in the above Figure 16 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0299] The training module 320 is specifically configured to obtain, based on the first sample image and the second sample image, a feature vector corresponding to the first sample image and a feature vector corresponding to the second sample image using the image matching model to be trained, wherein the first sample image and the second sample image belong to a positive sample image pair, and the positive sample image pair corresponds to a positive sample label;

[0300] Based on the first sample image and the third sample image, obtaining a feature vector corresponding to the first sample image and a feature vector corresponding to the third sample image through the image matching model to be trained, the first sample image and the third sample image belonging to a negative sample image pair, and the negative sample image pair corresponding to a negative sample label;

[0301] determining a first characteristic distance according to a characteristic vector corresponding to the first sample image and a characteristic vector corresponding to the second sample image;

[0302] determining a second characteristic distance according to a characteristic vector corresponding to the first sample image and a characteristic vector corresponding to the third sample image;

[0303] A second loss function is used to update model parameters of the image matching model to be trained according to the first feature distance, the second feature distance, the positive sample label, and the negative sample label.

[0304] The present application provides a model training device, which uses the above-mentioned device to use the network structure of the twin neural network as the network structure of the image matching model. Therefore, the contrast loss function can be used to effectively process the relationship between paired sample images in the twin neural network, and express the matching degree of paired sample images well, so that it can be used well for training feature extraction models.

[0305] Optionally, in the above Figure 16 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0306] The training module 320 is specifically configured to obtain, based on the first sample image, an embedding vector corresponding to the first sample image through the image matching model to be trained;

[0307] Based on the second sample image, obtaining an embedding vector corresponding to the second sample image through the image matching model to be trained, wherein the second sample image is a positive sample image;

[0308] Based on the third sample image, obtaining an embedding vector corresponding to the third sample image through the image matching model to be trained, wherein the third sample image is a negative sample image;

[0309] Determining a first embedding distance according to the embedding vector corresponding to the first sample image and the embedding vector corresponding to the second sample image;

[0310] determining a second embedding distance according to the embedding vector corresponding to the first sample image and the embedding vector corresponding to the third sample image;

[0311] According to the first embedding distance and the second embedding distance, a third loss function is used to update the model parameters of the image matching model to be trained.

[0312] The present application provides a model training device, which uses the above-mentioned device to use the network structure of the twin neural network as the network structure of the image matching model. Therefore, the triple loss function can be used to effectively process the relationship between the three associated sample images in the twin neural network, and well express the matching degree of the three associated sample images, thereby being well used for training the model for extracting features.

[0313] Optionally, in the above Figure 16 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0314] The training module 320 is specifically configured to obtain a predicted classification result by using the image matching model to be trained based on the first sample image and the second sample image, where the first sample image and the second sample image correspond to annotated classification labels;

[0315] According to the predicted classification results and the marked classification labels, the fourth loss function is used to update the model parameters of the image matching model to be trained.

[0316] The present application provides a model training device, which uses the above-mentioned device to use the network structure of the twin neural network as the network structure of the image matching model. Therefore, the cross-entropy loss function can be used to effectively process the relationship between paired sample images in the twin neural network, and well express the matching degree of paired sample images, thereby being well used to train a model for outputting image similarity scores.

[0317] Optionally, in the above Figure 16 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0318] The acquisition module 310 is further configured to acquire a training sample image, wherein the training sample image includes M pixels, each pixel corresponds to a category label, and the category label is used to indicate whether the pixel belongs to the region of interest or the background region, and M is an integer greater than 1;

[0319] The acquisition module 310 is further configured to acquire the category prediction probability of each pixel point through the semantic segmentation model to be trained based on the training sample image;

[0320] The training module 320 is further used to update the model parameters of the semantic segmentation model to be trained using the fifth loss function according to the category prediction probability of each pixel point and the category annotation label of each pixel point until the model training conditions are met to obtain the semantic segmentation model.

[0321] The present application provides a model training device. Using the above device, it can be trained to not only segment the region of interest and the background area from the image, but also segment the text area and picture area in the region of interest, so that the randomly sampled image blocks can cover more effective content. As a result, the image can be framed more specifically, and representative image blocks can be taken out as sample images, which is conducive to training a more robust image matching model.

[0322] The image matching device in this application is described in detail below. Figure 17 , Figure 17 This is a schematic diagram of an embodiment of an image matching device in an embodiment of the present application. The image matching device 40 includes:

[0323] An acquisition module 410 is configured to acquire a first image to be matched;

[0324] The acquisition module 410 is further configured to acquire a second image to be matched;

[0325] The determination module 420 is configured to determine a matching result based on the first image to be matched and the second image to be matched by using an image matching model, wherein the image matching model is trained using the method provided in the above embodiment.

[0326] This application provides an image matching device that uses the aforementioned device to divide an image into a region of interest and a background region. This allows for more targeted selection of randomly selected image blocks, extracting representative image blocks as sample images, thereby facilitating the training of a more robust image matching model. This reduces the occurrence of mismatches during model inference.

[0327] Optionally, in the above Figure 17 On the basis of the corresponding embodiment, in another embodiment of the image matching device 40 provided in the embodiment of the present application,

[0328] The determination module 420 is specifically configured to obtain a feature vector corresponding to the first image to be matched by using an image matching model based on the first image to be matched;

[0329] Based on the second image to be matched, obtaining a feature vector corresponding to the second image to be matched through an image matching model;

[0330] Determining a target feature distance based on a feature vector corresponding to the first image to be matched and a feature vector corresponding to the second image to be matched;

[0331] If the target feature distance is less than or equal to the distance threshold, the matching result is determined to be a successful match;

[0332] If the target feature distance is greater than the distance threshold, the matching result is determined to be a matching failure.

[0333] This application provides an image matching device. Using the aforementioned device, an image matching model directly outputs feature vectors of two images to be matched, and then determines the distance between features based on the feature vectors of the two images to be matched. Because a smaller distance between features indicates a higher degree of similarity, the image matching model can output feature vectors to determine the matching result of the two images to be matched, thereby improving the accuracy of image matching.

[0334] Optionally, in the above Figure 17 On the basis of the corresponding embodiment, in another embodiment of the image matching device 40 provided in the embodiment of the present application,

[0335] The determination module 420 is specifically configured to obtain a similarity score based on the first image to be matched and the second image to be matched by using an image matching model;

[0336] If the similarity score is greater than or equal to the similarity threshold, the matching result is determined to be a successful match;

[0337] If the similarity score is less than the similarity threshold, the matching result is determined to be a matching failure.

[0338] The present application provides an image matching device. Using the above-mentioned device, the image matching model directly outputs the similarity score between the two images to be matched. Since the larger the similarity score, the higher the degree of similarity, the similarity score can be directly output based on the image matching model to determine the matching result of the two images to be matched, thereby improving the accuracy of image matching.

[0339] The embodiment of the present application also provides another model training device and an image matching device, which can be deployed on a server. Figure 18 This is a schematic diagram of a server structure provided in an embodiment of the present application. The server 500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 522 (for example, one or more processors) and a memory 532, and one or more storage media 530 (for example, one or more massive storage devices) for storing application programs 542 or data 544. Among them, the memory 532 and the storage medium 530 can be temporary storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 522 can be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the server 500.

[0340] The server 500 may also include one or more power supplies 526, one or more wired or wireless network interfaces 550, one or more input and output interfaces 558, and / or one or more operating systems 541, such as Windows Server 2003 or Windows Server 2003R. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0341] The steps performed by the server in the above embodiment can be based on the Figure 18 The server structure shown.

[0342] The embodiment of the present application also provides another model training device and an image matching device, which can be deployed on a terminal device. Figure 19 For the sake of convenience, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. In the embodiment of the present application, the terminal device is a smartphone as an example for explanation:

[0343] Figure 19 The block diagram shows a partial structure of a smart phone related to the terminal device provided in the embodiment of the present application. Figure 19 The smartphone includes components such as a radio frequency (RF) circuit 610, a memory 620, an input unit 630, a display unit 640, a sensor 650, an audio circuit 660, a wireless fidelity (WiFi) module 670, a processor 680, and a power supply 690. It will be understood by those skilled in the art that Figure 19 The structure of the smartphone shown in the figure does not constitute a limitation on the smartphone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0344] The following combination Figure 19 A detailed introduction to the various components of a smartphone:

[0345] The RF circuit 610 can be used to receive and send signals during information transmission or calls. In particular, after receiving downlink information from the base station, it is sent to the processor 680 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 610 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 610 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to the global system of mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0346] The memory 620 can be used to store software programs and modules. The processor 680 executes the various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 620. The memory 620 may mainly include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the smartphone (such as audio data, a phone book, etc.). In addition, the memory 620 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0347] The input unit 630 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the smartphone. Specifically, the input unit 630 may include a touch panel 631 and other input devices 632. The touch panel 631, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using any suitable object or accessory such as a finger, stylus, etc. on or near the touch panel 631) and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 631 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction, detects the signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 680, and can receive commands sent by the processor 680 and execute them. In addition, the touch panel 631 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 631, the input unit 630 may further include other input devices 632. Specifically, the other input devices 632 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick.

[0348] The display unit 640 can be used to display information input by the user or information provided to the user and various menus of the smartphone. The display unit 640 may include a display panel 641. Optionally, the display panel 641 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 631 may cover the display panel 641. When the touch panel 631 detects a touch operation on or near it, it is transmitted to the processor 680 to determine the type of touch event. Subsequently, the processor 680 provides a corresponding visual output on the display panel 641 according to the type of touch event. Although in Figure 19 In the embodiment, the touch panel 631 and the display panel 641 are used as two independent components to realize the input and output functions of the smartphone, but in some embodiments, the touch panel 631 and the display panel 641 can be integrated to realize the input and output functions of the smartphone.

[0349] The smartphone may also include at least one sensor 650, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 641 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 641 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the smartphone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the smartphone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described in detail here.

[0350] Audio circuit 660, speaker 661, and microphone 662 provide an audio interface between the user and the smartphone. Audio circuit 660 converts received audio data into electrical signals and transmits them to speaker 661, which then converts them into sound signals for output. Microphone 662, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 660 and converted into audio data. The audio data is then processed by processor 680 and transmitted to, for example, another smartphone via RF circuit 610, or stored in memory 620 for further processing.

[0351] WiFi is a short-range wireless transmission technology. Smartphones can help users send and receive emails, browse the web, and access streaming media through the WiFi module 670. It provides users with wireless broadband Internet access. Figure 19 A WiFi module 670 is shown, but it is understandable that it is not an essential component of the smartphone and can be omitted as needed without changing the essence of the invention.

[0352] Processor 680 is the control center of the smartphone, connecting all components of the smartphone using various interfaces and circuits. It executes software programs and / or modules stored in memory 620 and accesses data stored in memory 620 to perform various smartphone functions and process data. Optionally, processor 680 may include one or more processing units. Alternatively, processor 680 may integrate an application processor and a modem processor, with the application processor primarily handling the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 680.

[0353] The smartphone also includes a power supply 690 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 680 through a power management system, thereby managing charging, discharging, and power consumption through the power management system.

[0354] Although not shown, the smartphone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0355] The steps performed by the terminal device in the above embodiment can be based on the Figure 19 The terminal device structure shown.

[0356] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0357] An embodiment of the present application also provides a computer program product including a program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0358] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0359] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0360] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0361] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0362] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0363] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: include: Obtaining a first image to be trained; Based on the first image to be trained, obtaining a first region segmentation result through a semantic segmentation model, wherein the first region segmentation result is used to determine a region of interest and a background region in the first image to be trained, the region of interest including at least one of a text region and a picture region; Acquiring a first sample image and a second sample image from the first to-be-trained image according to the first region segmentation result specifically includes: Randomly selecting a first image and a second image from the first to-be-trained images; determining, based on the first region segmentation result, an area ratio of the region of interest in the first image, and determining an area ratio of the region of interest in the second image; If the area ratio of the region of interest in the first image is greater than or equal to a ratio threshold, taking the first image as the first sample image; If the area ratio of the region of interest in the second image is greater than or equal to the ratio threshold, using the second image as the second sample image; The model parameters of the image matching model to be trained are updated according to the first sample image and the second sample image until a model training condition is met, thereby obtaining an image matching model.

2. The method according to claim 1, characterized in that The step of obtaining a first sample image and a second sample image derived from the first image to be trained according to the first region segmentation result further includes: Randomly selecting a first to-be-processed image and a second to-be-processed image from the first to-be-trained image; determining, based on the first region segmentation result, a region ratio of the region of interest in the first image to be processed, and determining a region ratio of the region of interest in the second image to be processed; If the area ratio of the region of interest in the first image to be processed is greater than or equal to the ratio threshold, performing data augmentation processing on the first image to be processed to obtain the first sample image; If the area ratio of the region of interest in the second image to be processed is greater than or equal to the ratio threshold, data augmentation processing is performed on the second image to be processed to obtain the second sample image.

3. The method according to claim 1, characterized in that The updating of model parameters of the image matching model to be trained according to the first sample image and the second sample image includes: Based on the first sample image, obtaining a first feature map through an encoding network included in the image matching model to be trained; Based on the first feature map, obtaining a first feature vector through a projection network included in the image matching model to be trained; Based on the first feature vector, obtaining a target feature vector through a prediction network included in the image matching model to be trained; Based on the second sample image, obtaining a second feature map through an encoding network included in the target model; Based on the second feature map, obtaining a second feature vector through a projection network included in the target model; According to the target feature vector and the second feature vector, the model parameters of the image matching model to be trained are updated by using a first loss function.

4. The method according to claim 1, wherein The method further comprises: Acquire a second image to be trained; Based on the second image to be trained, obtaining a second region segmentation result through the semantic segmentation model, wherein the second region segmentation result is used to determine a region of interest and a background region in the second image to be trained; Acquire, according to the second region segmentation result, a third sample image derived from the second image to be trained, wherein a proportion of the region of interest included in the third sample image is greater than or equal to a proportion threshold; The updating of the model parameters of the image matching model to be trained according to the first sample image and the second sample image until the model training conditions are met to obtain the image matching model includes: According to the first sample image, the second sample image, and the third sample image, the model parameters of the image matching model to be trained are updated until the model training conditions are met, thereby obtaining the image matching model.

5. The method according to claim 4, characterized in that The updating of the model parameters of the image matching model to be trained according to the first sample image, the second sample image, and the third sample image includes: Based on the first sample image and the second sample image, obtaining a feature vector corresponding to the first sample image and a feature vector corresponding to the second sample image through the image matching model to be trained, the first sample image and the second sample image being positive sample images, and the positive sample corresponding to a positive sample label; Based on the first sample image and the third sample image, obtaining a feature vector corresponding to the first sample image and a feature vector corresponding to the third sample image through the image matching model to be trained, wherein the first sample image and the third sample image are negative sample images, and the negative samples correspond to negative sample labels; determining a first characteristic distance according to a characteristic vector corresponding to the first sample image and a characteristic vector corresponding to the second sample image; determining a second characteristic distance according to the characteristic vector corresponding to the first sample image and the characteristic vector corresponding to the third sample image; According to the first feature distance, the second feature distance, the positive sample and the negative sample label, a second loss function is used to update the model parameters of the image matching model to be trained.

6. The method according to claim 4, characterized in that The updating of the model parameters of the image matching model to be trained according to the first sample image, the second sample image, and the third sample image includes: Based on the first sample image, obtaining an embedding vector corresponding to the first sample image through the image matching model to be trained; Based on the second sample image, obtaining an embedding vector corresponding to the second sample image through the image matching model to be trained, wherein the second sample image is a positive sample image; Based on the third sample image, obtaining an embedding vector corresponding to the third sample image through the image matching model to be trained, wherein the third sample image is a negative sample image; determining a first embedding distance according to the embedding vector corresponding to the first sample image and the embedding vector corresponding to the second sample image; determining a second embedding distance according to the embedding vector corresponding to the first sample image and the embedding vector corresponding to the third sample image; According to the first embedding distance and the second embedding distance, a third loss function is used to update the model parameters of the image matching model to be trained.

7. The method according to claim 1, characterized in that The updating of model parameters of the image matching model to be trained according to the first sample image and the second sample image includes: Based on the first sample image and the second sample image, obtaining a predicted classification result through the image matching model to be trained, the first sample image and the second sample image corresponding to the annotated classification label; According to the predicted classification result and the annotated classification label, a fourth loss function is used to update the model parameters of the image matching model to be trained.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Acquire a training sample image, wherein the training sample image includes M pixels, each pixel corresponds to a category label, the category label is used to indicate whether the pixel belongs to the region of interest or the background region, and M is an integer greater than 1; Based on the training sample image, obtaining the category prediction probability of each pixel point through the semantic segmentation model to be trained; According to the category prediction probability of each pixel point and the category annotation label of each pixel point, the fifth loss function is used to update the model parameters of the semantic segmentation model to be trained until the model training conditions are met to obtain the semantic segmentation model.

9. A method for image matching, characterized in that: include: Acquire a first image to be matched; Acquire a second image to be matched; Based on the first image to be matched and the second image to be matched, a matching result is determined by an image matching model, wherein the image matching model is trained using the method of any one of claims 1 to 8.

10. The method according to claim 9, characterized in that The determining a matching result by using an image matching model based on the first image to be matched and the second image to be matched includes: Based on the first image to be matched, obtaining a feature vector corresponding to the first image to be matched by using the image matching model; Based on the second image to be matched, obtaining a feature vector corresponding to the second image to be matched by using the image matching model; determining a target feature distance according to a feature vector corresponding to the first image to be matched and a feature vector corresponding to the second image to be matched; If the target feature distance is less than or equal to the distance threshold, determining that the matching result is a successful match; If the target feature distance is greater than the distance threshold, the matching result is determined to be a matching failure.

11. The method according to claim 9, characterized in that The determining a matching result by using an image matching model based on the first image to be matched and the second image to be matched includes: Based on the first image to be matched and the second image to be matched, obtaining a similarity score by using the image matching model; If the similarity score is greater than or equal to the similarity threshold, the matching result is determined to be a successful match; If the similarity score is less than the similarity threshold, the matching result is determined to be a matching failure.

12. A model training device, characterized in that: include: An acquisition module, configured to acquire a first image to be trained; The acquisition module is further configured to obtain a first region segmentation result based on the first image to be trained using a semantic segmentation model, wherein the first region segmentation result is used to determine a region of interest and a background region in the first image to be trained, wherein the region of interest includes at least one of a text region and a picture region; The acquisition module is further configured to acquire a first sample image and a second sample image from the first image to be trained based on the first region segmentation result, specifically comprising: randomly selecting a first image and a second image from the first image to be trained; determining, based on the first region segmentation result, an area ratio of the region of interest in the first image, and determining an area ratio of the region of interest in the second image; if the area ratio of the region of interest in the first image is greater than or equal to a ratio threshold, using the first image as the first sample image; and if the area ratio of the region of interest in the second image is greater than or equal to the ratio threshold, using the second image as the second sample image; The training module is used to update the model parameters of the image matching model to be trained according to the first sample image and the second sample image until the model training conditions are met to obtain the image matching model.

13. The device according to claim 12, characterized in that The acquisition module is further configured to: Randomly selecting a first to-be-processed image and a second to-be-processed image from the first to-be-trained image; determining, based on the first region segmentation result, a region ratio of the region of interest in the first image to be processed, and determining a region ratio of the region of interest in the second image to be processed; If the area ratio of the region of interest in the first image to be processed is greater than or equal to the ratio threshold, performing data augmentation processing on the first image to be processed to obtain the first sample image; If the area ratio of the region of interest in the second image to be processed is greater than or equal to the ratio threshold, data augmentation processing is performed on the second image to be processed to obtain the second sample image.

14. The device according to claim 12, characterized in that The training module is specifically used to: Based on the first sample image, obtaining a first feature map through an encoding network included in the image matching model to be trained; Based on the first feature map, obtaining a first feature vector through a projection network included in the image matching model to be trained; Based on the first feature vector, obtaining a target feature vector through a prediction network included in the image matching model to be trained; Based on the second sample image, obtaining a second feature map through an encoding network included in the target model; Based on the second feature map, obtaining a second feature vector through a projection network included in the target model; According to the target feature vector and the second feature vector, the model parameters of the image matching model to be trained are updated by using a first loss function.

15. The device according to claim 12, characterized in that The acquisition module is further configured to acquire a second image to be trained; The acquisition module is further configured to acquire a second region segmentation result based on the second image to be trained using the semantic segmentation model, wherein the second region segmentation result is used to determine a region of interest and a background region in the second image to be trained; The acquisition module is further configured to acquire, based on the second region segmentation result, a third sample image derived from the second image to be trained, wherein a proportion of the region of interest included in the third sample image is greater than or equal to a proportion threshold; The training module is specifically used to update the model parameters of the image matching model to be trained according to the first sample image, the second sample image and the third sample image until the model training conditions are met to obtain the image matching model.

16. The device according to claim 15, characterized in that The training module is specifically used to: Based on the first sample image and the second sample image, obtaining a feature vector corresponding to the first sample image and a feature vector corresponding to the second sample image through the image matching model to be trained, the first sample image and the second sample image being positive sample images, and the positive sample corresponding to a positive sample label; Based on the first sample image and the third sample image, obtaining a feature vector corresponding to the first sample image and a feature vector corresponding to the third sample image through the image matching model to be trained, wherein the first sample image and the third sample image are negative sample images, and the negative samples correspond to negative sample labels; determining a first characteristic distance according to a characteristic vector corresponding to the first sample image and a characteristic vector corresponding to the second sample image; determining a second characteristic distance according to the characteristic vector corresponding to the first sample image and the characteristic vector corresponding to the third sample image; According to the first feature distance, the second feature distance, the positive sample and the negative sample label, a second loss function is used to update the model parameters of the image matching model to be trained.

17. The device according to claim 15, characterized in that The training module is specifically used to: Based on the first sample image, obtaining an embedding vector corresponding to the first sample image through the image matching model to be trained; Based on the second sample image, obtaining an embedding vector corresponding to the second sample image through the image matching model to be trained, wherein the second sample image is a positive sample image; Based on the third sample image, obtaining an embedding vector corresponding to the third sample image through the image matching model to be trained, wherein the third sample image is a negative sample image; determining a first embedding distance according to the embedding vector corresponding to the first sample image and the embedding vector corresponding to the second sample image; determining a second embedding distance according to the embedding vector corresponding to the first sample image and the embedding vector corresponding to the third sample image; According to the first embedding distance and the second embedding distance, a third loss function is used to update the model parameters of the image matching model to be trained.

18. The device according to claim 12, characterized in that The training module is specifically used to: Based on the first sample image and the second sample image, obtaining a predicted classification result through the image matching model to be trained, the first sample image and the second sample image corresponding to the annotated classification label; According to the predicted classification result and the annotated classification label, a fourth loss function is used to update the model parameters of the image matching model to be trained.

19. The device according to any one of claims 12 to 18, characterized in that The acquisition module is further configured to acquire a training sample image, wherein the training sample image includes M pixels, each pixel corresponds to a category label, the category label is used to indicate whether the pixel belongs to the region of interest or the background region, and M is an integer greater than 1; The acquisition module is further configured to acquire the category prediction probability of each pixel point through the semantic segmentation model to be trained based on the training sample image; The training module is also used to update the model parameters of the semantic segmentation model to be trained using a fifth loss function according to the category prediction probability of each pixel point and the category annotation label of each pixel point until the model training conditions are met to obtain a semantic segmentation model.

20. An image matching device, characterized in that: include: An acquisition module, configured to acquire a first image to be matched; The acquisition module is further configured to acquire a second image to be matched; A determination module is used to determine a matching result based on the first image to be matched and the second image to be matched by an image matching model, wherein the image matching model is trained using the method described in any one of claims 1 to 8.

21. The device according to claim 20, characterized in that The determining module is specifically configured to: Based on the first image to be matched, obtaining a feature vector corresponding to the first image to be matched by using the image matching model; Based on the second image to be matched, obtaining a feature vector corresponding to the second image to be matched by using the image matching model; determining a target feature distance according to a feature vector corresponding to the first image to be matched and a feature vector corresponding to the second image to be matched; If the target feature distance is less than or equal to the distance threshold, determining that the matching result is a successful match; If the target feature distance is greater than the distance threshold, the matching result is determined to be a matching failure.

22. The device according to claim 20, characterized in that The determining module is specifically configured to: Based on the first image to be matched and the second image to be matched, obtaining a similarity score by using the image matching model; If the similarity score is greater than or equal to the similarity threshold, the matching result is determined to be a successful match; If the similarity score is less than the similarity threshold, the matching result is determined to be a matching failure.

23. A computer device, characterized in that: include: Memory, processor, and bus system; Wherein, the memory is used to store programs; The processor is configured to execute the program in the memory, and the processor is configured to execute the method according to any one of claims 1 to 8, or the method according to any one of claims 9 to 11 according to the instructions in the program code; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

24. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to execute the method of any one of claims 1 to 8, or the method of any one of claims 9 to 11.

25. A computer program product, characterized in that The method comprises computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of any one of claims 1 to 8, or executes the method of any one of claims 9 to 11.

Images