Image recognition method and device, training method and device, equipment and readable storage medium

Abstract

The embodiment of the invention provides an image recognition method and device, a training method and device, equipment and a readable storage medium, and relates to the fields of artificial intelligence, maps and the like, and application scenes include but are not limited to image recognition scenes. The method comprises the steps that based on an obtained target image and a preset domain sharing prompt, an image feature vector of the target image is determined, the preset domain sharing prompt is used for representing cross-domain invariant information, and the cross-domain invariant information comprises non-image apparent change image information; based on the image feature vector of the target image and the text feature vector of each candidate category text in a preset candidate category text set, determining the similarity between the image feature vector of the target image and the text feature vector of each candidate category text, each candidate category text being used for describing one image category; determining the image category of the target image based on the similarities; therefore, the cross-domain invariant information of the target image is extracted, and the recognition accuracy of the target image is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image recognition method, training method, apparatus, device, and readable storage medium. Background Art

[0002] In existing technologies, image content transformation is performed on an image to obtain a transformed image. Image corruption (apparent damage to an image) is a common form of image corruption, including translation, rotation, noise injection, blurring, and weather simulation. Simulated weather, for example, involves adding fog, snow, or rain. Existing technologies use traditional convolutional neural networks to perform image recognition on transformed images, but this often results in low image recognition accuracy. Summary of the Invention

[0003] In response to the shortcomings of existing methods, the present disclosure proposes an image recognition method, training method, device, equipment, computer-readable storage medium and computer program product to solve the problem of how to improve the accuracy of image recognition.

[0004] In a first aspect, the present disclosure provides an image recognition method, comprising:

[0005] Determining an image feature vector of the target image based on the acquired target image and a preset domain-sharing hint, wherein the preset domain-sharing hint is used to represent cross-domain invariant information, where the cross-domain invariant information includes image information other than image appearance changes;

[0006] Determining the similarity between the image feature vector of the target image and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the target image and the text feature vector of each candidate category text, each candidate category text being used to describe an image category;

[0007] Based on the similarities, the image category of the target image is determined.

[0008] In one embodiment, determining an image feature vector of the target image based on the acquired target image and a preset domain sharing hint includes:

[0009] Determine a preprocessed target image based on the acquired target image through preprocessing;

[0010] Based on the preprocessed target image, the preprocessed target image is segmented into a plurality of image blocks through segmentation processing;

[0011] Mapping each image block in the plurality of image blocks into a low-dimensional vector, and constructing each low-dimensional vector into an image label sequence corresponding to the plurality of image blocks;

[0012] An image feature vector of a target image is determined based on an image tag sequence corresponding to a plurality of image blocks and a plurality of learnable vectors in a preset domain-sharing hint, wherein each of the plurality of learnable vectors is used to represent a vector capable of gradient updating.

[0013] In one embodiment, determining an image feature vector of a target image based on image tag sequences corresponding to a plurality of image blocks and a plurality of learnable vectors in a preset domain-shared hint includes:

[0014] Determine a cascaded vector based on image tag sequences corresponding to the plurality of image blocks and a plurality of learnable vectors in a preset domain-shared hint through cascade processing;

[0015] Based on the vector after cascade processing, the image feature vector of the target image is obtained through the image encoder in the preset pre-training model.

[0016] In one embodiment, before determining the similarity between the image feature vector of the target image and the text feature vector of each candidate category text in the preset candidate category text set based on the image feature vector of the target image and the text feature vector of each candidate category text, the method further includes:

[0017] Get multiple image categories;

[0018] Based on the multiple image categories and the preset prompt word encapsulation, determining candidate category text sets corresponding to the multiple image categories, and determining the candidate category text sets corresponding to the multiple image categories as the preset candidate category text set;

[0019] Determine the word embedding vector group corresponding to the preset candidate category text set;

[0020] Based on the word embedding vector group, the text feature vector of each candidate category text in the preset candidate category text set is obtained through the text encoder in the preset pre-trained model.

[0021] In one embodiment, determining the image category of the target image based on the similarities includes:

[0022] The maximum similarity among the similarities is determined, and the image category described in the candidate category text corresponding to the maximum similarity is determined as the image category of the target image.

[0023] In one embodiment, before determining the image feature vector of the target image based on the acquired target image and the preset domain sharing hint, the method further includes:

[0024] Obtain source image samples in the source domain and target image samples in the target domain. The source domain is used to represent the source image sample set, and the target domain is used to represent the target image sample set. The target image samples are obtained by performing image content transformation on the source image samples.

[0025] Determining an image feature vector of the source image sample based on the source image sample and the obtained domain-shared cue to be trained, and determining an image feature vector of the target image sample based on the target image sample and the domain-shared cue to be trained, wherein each learnable vector in the domain-shared cue to be trained is used to represent a vector that can be gradient updated during the training process of the domain-shared cue to be trained;

[0026] Determining a first similarity between the image feature vector of the source image sample and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the source image sample and the text feature vector of each candidate category text, and determining a first similarity vector based on each first similarity, the first similarity vector being used to represent the set of each first similarity;

[0027] Determining a second similarity between the image feature vector of the target image sample and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the target image sample and the text feature vector of each candidate category text, and determining a second similarity vector based on each second similarity, the second similarity vector being used to represent the set of each second similarity;

[0028] Determining a value of a consistency constraint loss function and a value of a classification loss function based on the first similarity vector and the second similarity vector;

[0029] Based on the values ​​of the consistency constraint loss function and the classification loss function, the domain sharing prompt to be trained is updated until the values ​​of the consistency constraint loss function and the classification loss function meet predetermined conditions, and the domain sharing prompt to be trained is updated to obtain the trained domain sharing prompt.

[0030] In one embodiment, determining a value of a consistency constraint loss function and a value of a classification loss function based on the first similarity vector and the second similarity vector includes:

[0031] Determining a value of a consistency constraint loss function based on the first similarity vector and the second similarity vector, where the value of the consistency constraint loss function is used to characterize a degree of distribution difference between the first similarity vector and the second similarity vector;

[0032] Based on the first similarity vector, a value of a first classification loss function is determined, and based on the second similarity vector, a value of a second classification loss function is determined.

[0033] In one embodiment, based on the values ​​of the consistency constraint loss function and the classification loss function satisfying predetermined conditions, the domain sharing hint to be trained is updated to obtain the trained domain sharing hint, including:

[0034] If the sum of the values ​​of the consistency constraint loss function, the first classification loss function, and the second classification loss function is equal to or less than the preset loss threshold, the domain sharing prompt to be trained is updated based on the values ​​of the consistency constraint loss function, the first classification loss function, and the second classification loss function to obtain the trained domain sharing prompt.

[0035] In a second aspect, the present disclosure provides a training method, comprising:

[0036] Obtain source image samples in the source domain and target image samples in the target domain. The source domain is used to represent the source image sample set, and the target domain is used to represent the target image sample set. The target image samples are obtained by performing image content transformation on the source image samples.

[0037] Determining the value of the consistency constraint loss function and the value of the classification loss function based on the source image sample, the target image sample, the preset candidate category text set, and the obtained domain shared hint to be trained;

[0038] Based on the values ​​of the consistency constraint loss function and the classification loss function, the domain sharing prompt to be trained is updated until the values ​​of the consistency constraint loss function and the classification loss function meet predetermined conditions, and the domain sharing prompt to be trained is updated to obtain the trained domain sharing prompt.

[0039] In one embodiment, based on the source image sample, the target image sample, the preset candidate category text set, and the obtained domain-shared hint to be trained, determining the value of the consistency constraint loss function and the value of the classification loss function includes:

[0040] Determining an image feature vector of the source image sample based on the source image sample and the obtained domain-shared cue to be trained, and determining an image feature vector of the target image sample based on the target image sample and the domain-shared cue to be trained, wherein each learnable vector in the domain-shared cue to be trained is used to represent a vector that can be gradient updated during the training process of the domain-shared cue to be trained;

[0041] Determining a first similarity between the image feature vector of the source image sample and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the source image sample and the text feature vector of each candidate category text, and determining a first similarity vector based on each first similarity, the first similarity vector being used to represent the set of each first similarity;

[0042] Determining a second similarity between the image feature vector of the target image sample and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the target image sample and the text feature vector of each candidate category text, and determining a second similarity vector based on each second similarity, the second similarity vector being used to represent the set of each second similarity;

[0043] Based on the first similarity vector and the second similarity vector, a value of a consistency constraint loss function and a value of a classification loss function are determined.

[0044] In one embodiment, determining a value of a consistency constraint loss function and a value of a classification loss function based on the first similarity vector and the second similarity vector includes:

[0045] Determining a value of a consistency constraint loss function based on the first similarity vector and the second similarity vector, where the value of the consistency constraint loss function is used to characterize a degree of distribution difference between the first similarity vector and the second similarity vector;

[0046] Based on the first similarity vector, a value of a first classification loss function is determined, and based on the second similarity vector, a value of a second classification loss function is determined.

[0047] In a third aspect, the present disclosure provides an image recognition device, comprising:

[0048] a first processing module, configured to determine an image feature vector of the target image based on the acquired target image and a preset domain sharing hint, wherein the preset domain sharing hint is used to represent cross-domain invariant information, where the cross-domain invariant information includes image information other than image appearance changes;

[0049] a second processing module, configured to determine a similarity between the image feature vector of the target image and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the target image and the text feature vector of each candidate category text, wherein each candidate category text is used to describe an image category;

[0050] The third processing module is configured to determine the image category of the target image based on the similarities.

[0051] In a fourth aspect, the present disclosure provides a training device, comprising:

[0052] a fourth processing module, configured to obtain source image samples in a source domain and target image samples in a target domain, wherein the source domain is used to represent a set of source image samples, and the target domain is used to represent a set of target image samples, and the target image samples are obtained by performing image content transformation on the source image samples;

[0053] Determining the value of the consistency constraint loss function and the value of the classification loss function based on the source image sample, the target image sample, the preset candidate category text set, and the obtained domain shared hint to be trained;

[0054] Based on the values ​​of the consistency constraint loss function and the classification loss function, the domain sharing prompt to be trained is updated until the values ​​of the consistency constraint loss function and the classification loss function meet predetermined conditions, and the domain sharing prompt to be trained is updated to obtain the trained domain sharing prompt.

[0055] In a fifth aspect, the present disclosure provides an electronic device, comprising: a processor, a memory, and a bus;

[0056] Bus, used to connect the processor and memory;

[0057] A memory, for storing operation instructions;

[0058] The processor is used to execute the image recognition method of the first aspect of the present disclosure or the training method of the second aspect of the present disclosure by calling an operation instruction.

[0059] In a sixth aspect, the present disclosure provides a computer-readable storage medium storing a computer program, which is used to execute the image recognition method of the first aspect of the present disclosure or the training method of the second aspect of the present disclosure.

[0060] In a seventh aspect, the present disclosure provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the image recognition method in the first aspect of the present disclosure or the training method in the second aspect of the present disclosure.

[0061] The technical solutions provided by the embodiments of the present disclosure have at least the following beneficial effects:

[0062] Based on the acquired target image and the preset domain sharing hint, the image feature vector of the target image is determined. The preset domain sharing hint is used to represent cross-domain invariant information, which includes image information that does not change in image appearance. Based on the image feature vector of the target image and the text feature vector of each candidate category text in the preset candidate category text set, the similarity between the image feature vector of the target image and the text feature vector of each candidate category text is determined. Each candidate category text is used to describe an image category. Based on each similarity, the image category of the target image is determined. In this way, based on the preset domain sharing hint (the trained domain sharing hint), the cross-domain invariant information of the target image is extracted during the target image recognition process, the recognition robustness of the target image (for example, the target image is obtained through image content transformation) is enhanced when dealing with different image content transformations, and the accuracy of image recognition for the target image is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for describing the embodiments of the present disclosure.

[0064] Figure 1 A schematic diagram of the architecture of an image recognition system provided in an embodiment of the present disclosure;

[0065] Figure 2 A flowchart of an image recognition method provided by an embodiment of the present disclosure;

[0066] Figure 3 A schematic diagram of image recognition provided by an embodiment of the present disclosure;

[0067] Figure 4 A schematic diagram of image recognition provided by an embodiment of the present disclosure;

[0068] Figure 5 A flowchart of an image recognition method provided by an embodiment of the present disclosure;

[0069] Figure 6 A schematic structural diagram of an image recognition device provided in an embodiment of the present disclosure;

[0070] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0071] The following describes embodiments of the present disclosure in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions of the embodiments of the present disclosure.

[0072] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present disclosure mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements, and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components, and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to the connection relationship between the element and the other element established through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" indicates implementation as "A", or implementation as "B", or implementation as "A and B".

[0073] It is understandable that in the specific implementation of the present disclosure, data related to image recognition is involved. When the above embodiments of the present disclosure are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0074] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0075] The embodiment of the present disclosure is an image recognition method provided by an image recognition system, which relates to fields such as artificial intelligence and maps.

[0076] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0077] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0078] Intelligent Traffic System (ITS), also known as Intelligent Transportation System, effectively integrates advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control and vehicle manufacturing, strengthens the connection between vehicles, roads and users, and thus forms a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy.

[0079] In order to better understand and illustrate the solutions of the embodiments of the present disclosure, some technical terms involved in the embodiments of the present disclosure are briefly explained below.

[0080] CLIP: CLIP (Contrastive Language-Image Pretrain) is a large-scale text-image cross-modal pretraining model based on noisy contrastive learning. CLIP has strong discriminative capabilities in the image field.

[0081] Prompt Learning: Prompt Learning is a common method in the field of natural language processing. It uses a common text template to encapsulate the input text, thereby significantly reducing the modal difference between the input text and the training data used by the pre-trained language model, thereby improving the transfer effect. In the field of computer vision, prompt learning is usually based on CLIP, setting "prompts" for text input or image input and encapsulating it, thereby improving the transfer effect of weakly supervised image-text models on downstream visual tasks.

[0082] Source domain and target domain: The main forms of image content transformation include image corruption, such as image translation, image rotation, noise injection, image blurring, and weather simulation. Weather simulation in images includes adding fog, snow, rain, etc.; the ImageNet-C dataset includes 15 types of image corruption, and each type of image corruption contains 5 levels of image content transformation intensity {1, 2, 3, 4, 5}; for example, the original image (the image without any image corruption) is the source domain image in the source domain, and the image in the ImageNet-C dataset (the image with any type of image corruption or any intensity of image corruption) is the target domain image in the target domain.

[0083] Pre-trained model: A pre-trained model is a general model trained on a large-scale dataset. The pre-trained model learns general knowledge through pre-training and then fine-tunes it on specific tasks to adapt to different scenarios and needs.

[0084] RGB: The RGB color model is a color standard in the industry. It obtains various colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other. RGB represents the colors of the three channels of red, green, and blue. This standard covers almost all colors that can be perceived by human vision and is one of the most widely used color systems.

[0085] BPE algorithm: The main purpose of the BPE (Byte Pair Encoder) algorithm is data compression. The BPE algorithm is described as an iterative process in which the most frequent pair of characters in a string is replaced by a character that does not appear in this string.

[0086] The solutions provided by the embodiments of the present disclosure relate to image recognition technology. The technical solutions of the present disclosure are described in detail below using specific embodiments. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present disclosure will be described below in conjunction with the accompanying drawings.

[0087] In order to better understand the solution provided by the embodiment of the present disclosure, the solution is described below in conjunction with a specific application scenario.

[0088] In one embodiment, Figure 1FIG. 1 shows a schematic diagram of an image recognition system architecture applicable to an embodiment of the present disclosure. It can be understood that the image recognition method provided by the embodiment of the present disclosure can be applied to, but not limited to, Figure 1 In the application scenario shown.

[0089] In this example, Figure 1 As shown, the architecture of the image recognition system in this example may include but is not limited to a server 10, a terminal 20, and a database 30. The server 10, the terminal 20, and the database 30 may interact with each other via a network 40.

[0090] Based on the acquired target image and a preset domain sharing hint, the server 10 determines an image feature vector of the target image. The preset domain sharing hint is used to represent cross-domain invariant information, which includes image information that does not change in image appearance. Based on the image feature vector of the target image and the text feature vector of each candidate category text in a preset candidate category text set, the server 10 determines the similarity between the image feature vector of the target image and the text feature vector of each candidate category text, each candidate category text being used to describe an image category. Based on the respective similarities, the server 10 determines the image category of the target image. The server 10 transmits the image category of the target image to the terminal 20 for display and transmits the image category of the target image to the database 30 for storage.

[0091] It is understood that the above is only an example and is not limited to this embodiment.

[0092] Among them, terminals include but are not limited to smartphones (such as Android phones, iOS phones, etc.), mobile phone simulators, tablet computers, laptops, digital broadcast receivers, MIDs (Mobile Internet Devices), PDAs (Personal Digital Assistants), intelligent voice interaction devices, smart home appliances, car terminals, etc.

[0093] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0094] Cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network that provides these resources is called the "cloud." To users, these resources appear infinitely scalable and can be accessed at any time, used on demand, expanded at any time, and paid for on a per-use basis.

[0095] As a provider of cloud computing infrastructure, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose and use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0096] Based on logical functional divisions, the PaaS (Platform as a Service) layer can be deployed on top of the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. SaaS can also be deployed directly on top of IaaS. PaaS is a platform for software execution, such as databases and web containers. SaaS is a variety of business software, such as web portals and text messaging apps. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0097] Artificial intelligence cloud services, also known as AIaaS (AI as a Service), are a mainstream AI platform offering. Specifically, AIaaS platforms break down several common AI services and provide them as standalone or packaged services in the cloud. This service model is similar to an AI-themed marketplace: all developers can access one or more of the platform's AI services through APIs. Experienced developers can also use the platform's AI framework and infrastructure to deploy and maintain their own cloud AI services.

[0098] The aforementioned networks may include, but are not limited to, wired networks and wireless networks. Wired networks include local area networks, metropolitan area networks, and wide area networks, and wireless networks include Bluetooth, Wi-Fi, and other wireless communication networks. The specific network type may be determined based on actual application scenarios and is not limited here.

[0099] See also Figure 2 , Figure 2 The flowchart of an image recognition method provided by an embodiment of the present disclosure is shown, wherein the method can be executed by any electronic device, such as a server, etc. As an optional implementation, the method can be executed by a server. For the convenience of description, in the description of some optional embodiments below, the server will be used as an example to illustrate the execution subject of the method. Figure 2 As shown, the image recognition method provided by the embodiment of the present disclosure includes the following steps:

[0100] S201, determining an image feature vector of the target image based on an acquired target image and a preset domain sharing hint, wherein the preset domain sharing hint is used to represent cross-domain invariant information, and the cross-domain invariant information includes image information other than image appearance changes.

[0101] Specifically, the target image is, for example, a source domain image or a target domain image, where the source domain image is the original image, and the original image is transformed through image content transformation to obtain the target domain image; image content transformation is, for example, Image Corruption, where Image Corruption includes image translation, image rotation, noise injection into the image, image blurring, and weather simulation in the image, where weather simulation in the image includes adding fog, snow, rain, etc. to the image; for example, the source domain image is transformed through Image Corruption, where the image appearance of the source domain image changes to obtain the target domain image.

[0102] The preset domain sharing hint is a trained domain sharing hint, and the number of the preset domain sharing hints can be at least one preset domain sharing hint. For example, a domain sharing hint to be trained is obtained; the domain sharing hint to be trained is trained to determine the trained domain sharing hint, and the trained domain sharing hint is determined as the preset domain sharing hint. For example, a preset domain sharing hint is a set of learnable vectors [p0, p1, ..., p n ], where n represents the number of learnable vectors, n is a positive integer, p i The dimension is 1×768, 768 represents the dimension of image features, p i The value range of i is [0,n].

[0103] Cross-domain invariant information includes image information that does not change in image appearance, such as edges, objects, geometric structures, etc.; for example, an original image of a dog sitting on the grass, the original image is the source domain image, and noise (such as Gaussian noise) is injected into the original image, and the original image becomes blurred from clear to obtain the target domain image, that is, the image appearance changes, the original image is a clear image, and the target domain image is a blurred image, but the object (such as dog, grass, etc.) and geometric structure (dog sitting on the grass) in the original image do not change, that is, the target domain image and the original image (source domain image) both have the same cross-domain invariant information.

[0104] For example, Figure 3 As shown, the preset domain sharing hint is the preset domain sharing hint 301, and the target image is the target image 302; based on the target image 302 and the preset domain sharing hint 301, the image feature vector 307 of the target image is determined by the image encoder 303 in CLIP.

[0105] S202, based on the image feature vector of the target image and the text feature vector of each candidate category text in a preset candidate category text set, determine the similarity between the image feature vector of the target image and the text feature vector of each candidate category text, each candidate category text is used to describe an image category.

[0106] Specifically, for example, the ImageNet-C dataset includes 1000 categories, that is, C = 1000. Prompt word encapsulation is used for each of the C categories to obtain C text descriptions, that is, C candidate category texts. The C candidate category texts are constructed into a preset candidate category text set; prompt word encapsulation, such as static Prompt encapsulation, static Prompt encapsulation, such as "a photo of a_____", candidate category text such as a photo of a dog, dog is a category, and candidate category text aphotoof a dog is used to describe the category of the image as dog.

[0107] For example, Figure 3 Described, based on the candidate category text set 304 (preset candidate category text set), the text feature vector 306 of the candidate category text (the text feature vector of each candidate category text in the preset candidate category text set) is determined through the text encoder 305 in CLIP; based on the image feature vector 307 of the target image and the text feature vector 306 of the candidate category text, the similarity between the image feature vector 307 of the target image and the text feature vector 306 of the candidate category text is determined, that is, the similarity between the image feature vector of the target image and the text feature vector of each candidate category text is determined.

[0108] S203: Determine the image category of the target image based on the similarities.

[0109] Specifically, for example, the preset candidate category text set includes 1,000 candidate category texts, and the image category of the target image is determined based on the similarity between the image feature vector of the target image and the text feature vectors of the 1,000 candidate category texts, that is, based on 1,000 similarities.

[0110] For example, Figure 3 As shown, based on the respective similarities, the image category 308 of the target image is determined.

[0111] In the embodiment of the present disclosure, based on the acquired target image and the preset domain sharing prompt, the image feature vector of the target image is determined, the preset domain sharing prompt is used to characterize cross-domain invariant information, and the cross-domain invariant information includes image information of non-image appearance changes; based on the image feature vector of the target image and the text feature vector of each candidate category text in the preset candidate category text set, the similarity between the image feature vector of the target image and the text feature vector of each candidate category text is determined, each candidate category text is used to describe an image category; based on each similarity, the image category of the target image is determined. In this way, based on the preset domain sharing prompt (the domain sharing prompt after training), in the process of target image recognition, the extraction of cross-domain invariant information of the target image is realized, the recognition robustness of the target image (for example, the target image is obtained by image content transformation) when responding to different image content transformations is enhanced, and the accuracy of image recognition for the target image is improved.

[0112] In one embodiment, determining an image feature vector of the target image based on the acquired target image and a preset domain sharing hint includes steps A1-A4:

[0113] Step A1: Based on the acquired target image, a preprocessed target image is determined through preprocessing.

[0114] Specifically, based on the target image, an image size-processed target image is determined through image size processing; based on the image size-processed target image, a pre-processed target image is determined through normalization processing.

[0115] For example, the target image is resized (the size of the target image is changed) to a resolution of 224×224 to obtain the target image after image size processing; the target image after image size processing is normalized, that is, the RGB value of each pixel in the target image after image size processing is subtracted from the mean value corresponding to the RGB channel to obtain a difference result, and the difference result is divided by the standard deviation corresponding to the RGB channel, so that each pixel value in the target image after image size processing is scaled to between -1 and 1 to obtain the preprocessed target image.

[0116] Step A2: Based on the pre-processed target image, the pre-processed target image is segmented into a plurality of image blocks through segmentation.

[0117] Specifically, for example, the preprocessed target image is input into the PatchEmbedding layer (image block embedding layer) of the image encoder in CLIP, and the preprocessed target image is divided into non-overlapping Patches, i.e., multiple image blocks, through the Patch Embedding layer.

[0118] Step A3: Map each of the multiple image blocks into a low-dimensional vector, and construct each low-dimensional vector into an image label sequence corresponding to the multiple image blocks.

[0119] Specifically, for example, Patches (multiple image blocks) are mapped into visual token sequences (image tokens) through the 3x3 convolution of the Patch Embedding layer. The visual token sequence is an image tag sequence corresponding to multiple image blocks.

[0120] For example, Patches is 9 patches, and each patch (each image block) in the 9 patches (multiple image blocks) is encoded through a convolutional layer to obtain a token (a low-dimensional vector), that is, each image block in the multiple image blocks is mapped to a low-dimensional vector, and a total of 9 tokens corresponding to the 9 patches are obtained. The 9 tokens are constructed into a visual token sequence, that is, an image label sequence corresponding to multiple image blocks.

[0121] In step A4, an image feature vector of the target image is determined based on the image label sequences corresponding to the multiple image blocks and multiple learnable vectors in the preset domain sharing prompt, where each of the multiple learnable vectors is used to represent a vector that can be gradient updated.

[0122] Specifically, the preset domain sharing prompt is a trained domain sharing prompt, and the number of preset domain sharing prompts can be at least one preset domain sharing prompt. For example, a domain sharing prompt to be trained is obtained; the domain sharing prompt to be trained is trained to determine the trained domain sharing prompt, and the trained domain sharing prompt is determined as the preset domain sharing prompt. For example, a preset domain sharing prompt is a set of learnable vectors [p0, p1, ..., p n ], a set of learnable vectors [p0,p1,...,p n ] are multiple learnable vectors; where n represents the number of learnable vectors, n is a positive integer, and p i The dimension is 1×768, 768 represents the dimension of image features, pi The value range of i is [0,n].

[0123] For example, the image tag sequence (visual token sequence) corresponding to multiple image blocks and the multiple learnable vectors in the preset domain sharing prompt (a set of learnable vectors [p0, p1, ..., p n ]) to cascade and obtain the cascaded vector. For example, the dimension of the visual token sequence is 196×768, and a set of learnable vectors [p0,p1,...,p n ] is of dimension 2×768, a visual token sequence and a set of learnable vectors [p0,p1,...,p n ] are cascaded to obtain a cascaded vector, the dimension of the cascaded vector is (196+2)×768, that is, the cascaded vector is a high-dimensional vector with a dimension of 198×768.

[0124] For example, based on the vector after cascade processing, the image features of the target image, that is, the image feature vector of the target image, are obtained through the Transformer network of the image encoder in CLIP.

[0125] In one embodiment, determining an image feature vector of a target image based on image tag sequences corresponding to a plurality of image blocks and a plurality of learnable vectors in a preset domain-shared hint comprises steps B1-B2:

[0126] Step B1: Determine a cascaded vector based on image tag sequences corresponding to multiple image blocks and multiple learnable vectors in preset domain-shared hints through cascade processing.

[0127] Specifically, for example, the image tag sequence (visual token sequence) corresponding to the multiple image blocks and the multiple learnable vectors in the preset domain sharing prompt (a set of learnable vectors [p0, p1, ..., p n ]) to cascade and obtain the cascaded vector. For example, the dimension of the visual token sequence is 196×768, and a set of learnable vectors [p0,p1,...,p n ] is of dimension 2×768, a visual token sequence and a set of learnable vectors [p0,p1,...,p n ] are cascaded to obtain a cascaded vector, the dimension of the cascaded vector is (196+2)×768, that is, the cascaded vector is a high-dimensional vector with a dimension of 198×768.

[0128] Step B2: Based on the vector after cascade processing, the image feature vector of the target image is obtained through the image encoder in the preset pre-training model.

[0129] Specifically, a preset pre-trained model such as CLIP is used to obtain the image features of the target image, that is, the image feature vector of the target image, based on the vector after cascade processing through the Transformer network of the image encoder in CLIP.

[0130] In one embodiment, before determining the similarity between the image feature vector of the target image and the text feature vector of each candidate category text in the preset candidate category text set based on the image feature vector of the target image and the text feature vector of each candidate category text, steps C1-C4 are further included:

[0131] Step C1: Acquire multiple image categories.

[0132] Specifically, for example, a variety of image categories are obtained from the ImageNet-C dataset, which includes 1,000 categories (image categories).

[0133] Step C2: determining candidate category text sets corresponding to the multiple image categories based on the multiple image categories and the preset prompt word encapsulation, and determining the candidate category text sets corresponding to the multiple image categories as the preset candidate category text set.

[0134] Specifically, for example, the ImageNet-C dataset includes 1000 categories (image categories), that is, C = 1000. For the C categories, preset prompt word encapsulation is used to obtain C text descriptions, that is, C candidate category texts, and the C candidate category texts are constructed into a preset candidate category text set; the preset prompt word encapsulation is, for example, a static Prompt encapsulation, and the static Prompt encapsulation is, for example, "aphoto of a_____", and the candidate category text is, for example, a photo of a cat, where cat is a category, and the candidate category text a photo of a cat is used to describe the category of the image as cat.

[0135] Step C3: Determine the word embedding vector group corresponding to the preset candidate category text set.

[0136] Specifically, for example, each candidate category text in the preset candidate category text set is a text of variable length; through word embedding processing, each variable-length text is converted into a word embedding vector of fixed dimension, that is, each candidate category text in the preset candidate category text set is converted into a word embedding vector, and all candidate category texts in the preset candidate category text set are converted into multiple word embedding vectors, and multiple word embedding vectors are constructed into a word embedding vector group (text tokens), that is, a word embedding vector group corresponding to the preset candidate category text set; wherein, in word embedding processing, such as the BPE algorithm, the dimension of text tokens is C×77×512, 77 represents the length of the word embedding vector, 512 is the text feature dimension of CLIP, and C represents the number of categories.

[0137] Step C4: Based on the word embedding vector group, a text feature vector of each candidate category text in the preset candidate category text set is obtained through a text encoder in a preset pre-trained model.

[0138] Specifically, a preset pre-trained model such as CLIP and a word embedding vector group such as text tokens are used to input texttokens into the Transformer network of the text encoder in CLIP to obtain the text features of each candidate category text in the preset candidate category text set, that is, the text feature vector of each candidate category text.

[0139] In one embodiment, determining the image category of the target image based on the similarities includes:

[0140] The maximum similarity among the similarities is determined, and the image category described in the candidate category text corresponding to the maximum similarity is determined as the image category of the target image.

[0141] Specifically, for example, the preset candidate category text set includes 1,000 candidate category texts, and the image category of the target image is determined based on the similarity between the image feature vector of the target image and the text feature vectors of the 1,000 candidate category texts, that is, based on the 1,000 similarities; for example, the image category described in the candidate category text corresponding to the maximum similarity among these 1,000 similarities is determined as the image category of the target image; for example, if the image category described in the candidate category text corresponding to the maximum similarity is dog, then the image category of the target image is determined to be dog.

[0142] In one embodiment, before determining the image feature vector of the target image based on the acquired target image and the preset domain sharing hint, steps D1-D6 are further included:

[0143] In step D1, source image samples in a source domain and target image samples in a target domain are obtained. The source domain is used to represent a set of source image samples, and the target domain is used to represent a set of target image samples. The target image samples are obtained by performing image content transformation on the source image samples.

[0144] Specifically, the source image samples in the source domain are source domain images, and the target image samples in the target domain are target domain images; the source image samples are transformed by image content to obtain target image samples.

[0145] Step D2: Determine the image feature vector of the source image sample based on the source image sample and the obtained domain-shared hint to be trained, and determine the image feature vector of the target image sample based on the target image sample and the domain-shared hint to be trained, where each learnable vector in the domain-shared hint to be trained is used to represent a vector that can be gradient updated during the training process of the domain-shared hint to be trained.

[0146] Specifically, for example, Figure 4 As shown, based on the source domain image 402 (source image sample) and the domain sharing prompt 401 (domain sharing prompt to be trained) in the source domain, the image feature vector 408 of the source domain image (source image sample) is determined; and based on the target domain image 403 (target image sample) and the domain sharing prompt 401 (domain sharing prompt to be trained) in the target domain, the image feature vector 409 of the target domain image (target image sample) is determined.

[0147] For example, Figure 4 As shown, based on the source domain image 402 (source image sample) in the source domain, the preprocessed source domain image (source image sample) is determined through preprocessing; based on the preprocessed source domain image (source image sample), the preprocessed source domain image (source image sample) is segmented into multiple image blocks through segmentation processing; each image block in the multiple image blocks is mapped to a low-dimensional vector, and each low-dimensional vector is constructed into an image label sequence corresponding to the multiple image blocks; based on the image label sequence corresponding to the multiple image blocks and the multiple learnable vectors in the domain shared prompt 401 (domain shared prompt to be trained), the cascaded processed vector is determined through cascade processing; based on the cascaded processed vector, the image feature vector 408 of the source domain image (source image sample) is obtained through the image encoder 404 in CLIP.

[0148] For example, as shown in the figure, based on the target domain image 403 (target image sample) in the target domain, the preprocessed target domain image (target image sample) is determined through preprocessing; based on the preprocessed target domain image (target image sample), the preprocessed target domain image (target image sample) is segmented into multiple image blocks through segmentation processing; each image block in the multiple image blocks is mapped to a low-dimensional vector, and each low-dimensional vector is constructed into an image label sequence corresponding to the multiple image blocks; based on the image label sequence corresponding to the multiple image blocks and the multiple learnable vectors in the domain shared prompt 401 (domain shared prompt to be trained), the cascaded processed vector is determined through cascade processing; based on the cascaded processed vector, the image feature vector 409 of the target domain image (target image sample) is obtained through the image encoder 404 in CLIP.

[0149] It should be noted that the image encoder 404 in CLIP processes the source domain image 402 in the source domain and the target domain image 403 in the target domain respectively; for example, after the image encoder 404 in CLIP has completed processing the source domain image 402 in the source domain, it then processes the target domain image 403 in the target domain; for another example, after the image encoder 404 in CLIP has completed processing the target domain image 403 in the target domain, it then processes the source domain image 402 in the source domain.

[0150] Step D3, based on the image feature vector of the source image sample and the text feature vector of each candidate category text in the preset candidate category text set, determines the first similarity between the image feature vector of the source image sample and the text feature vector of each candidate category text, and based on each first similarity, determines a first similarity vector, the first similarity vector being used to characterize the set of each first similarity.

[0151] Specifically, for example, Figure 4 As shown, multiple image categories are obtained; based on the multiple image categories and prompt word encapsulation, the candidate category text sets corresponding to the multiple image categories are determined, and the candidate category text sets corresponding to the multiple image categories are determined as the preset candidate category text sets; the word embedding vector group (text tokens) corresponding to the preset candidate category text set (candidate category text set 405) is determined; based on the word embedding vector group, the text feature vector of each candidate category text in the preset candidate category text set is obtained through the text encoder 406 in CLIP, that is, the text feature vector 407 of the candidate category text.

[0152] A similarity vector is a set of similarities, for example, a first similarity vector is a set of first similarities. Figure 4As shown, based on the image feature vector 408 of the source domain image (source image sample) and the text feature vector 407 of the candidate category text (the text feature vector of each candidate category text), the first similarity between the image feature vector 408 of the source domain image and the text feature vector 407 of the candidate category text is determined.

[0153] Step D4, based on the image feature vector of the target image sample and the text feature vector of each candidate category text in the preset candidate category text set, determines the second similarity between the image feature vector of the target image sample and the text feature vector of each candidate category text, and based on each second similarity, determines a second similarity vector, the second similarity vector being used to characterize the set of each second similarity.

[0154] Specifically, the similarity vector is a set of similarities, for example, the second similarity vector is a set of second similarities. Figure 4 As shown, based on the image feature vector 409 of the target domain image (target image sample) and the text feature vector 407 of the candidate category text (the text feature vector of each candidate category text), the second similarity between the image feature vector 409 of the target domain image and the text feature vector 407 of the candidate category text is determined.

[0155] Step D5: determining a value of a consistency constraint loss function and a value of a classification loss function based on the first similarity vector and the second similarity vector.

[0156] Specifically, consistency constraint loss functions include KL divergence, distance metric functions, etc., and distance metric functions include L1 loss function, L2 loss function, Smooth L1 loss function, etc. Classification loss functions include cross entropy loss function, BCE (Binary Cross Entropy) loss function, ASL (Adaptive Sample Loss Function), Focal loss (dynamically scaled cross entropy loss function), etc. For example, Figure 4 As shown, the consistency constraint loss function corresponds to the consistency constraint loss 412 , and the classification loss function corresponds to the classification loss 411 .

[0157] For example, based on the first similarity vector and the second similarity vector, a value of the consistency constraint loss function is determined; based on the first similarity vector, a value of the first classification loss function is determined; and based on the second similarity vector, a value of the second classification loss function is determined.

[0158] It should be noted that the distribution difference between the first similarity vector and the second similarity vector is minimized through the consistency constraint loss function, so as to mine the consistency information between the source domain and the target domain (such as cross-domain invariant information), that is, the feature expression of the target domain should be as close as possible to the feature expression of the source domain; through two cross entropy loss functions (the first classification loss function and the second classification loss function), the classification results of the similarity vector of the source domain (the first similarity vector) and the classification results of the similarity vector of the target domain (the second similarity vector) are constrained respectively.

[0159] In step D6, the domain sharing prompt to be trained is updated based on the value of the consistency constraint loss function and the value of the classification loss function; until the value of the consistency constraint loss function and the value of the classification loss function meet the predetermined conditions, the domain sharing prompt to be trained is updated to obtain the trained domain sharing prompt.

[0160] Specifically, if the value of the consistency constraint loss function and the value of the classification loss function do not meet the predetermined conditions, the domain sharing prompt to be trained is updated and steps D1-D6 are repeated; if the value of the consistency constraint loss function and the value of the classification loss function meet the predetermined conditions, the training of the domain sharing prompt to be trained is stopped, and the domain sharing prompt to be trained is updated to obtain the trained domain sharing prompt.

[0161] It should be noted that based on the fixed CLIP, the domain-shared cues to be trained are trained by constraining the consistency of the feature expressions of the source domain and the target domain (consistency constraint loss function), so that CLIP has recognition robustness for both the source domain and the target domain; feature alignment of the source domain image and the target domain image is achieved, and domain invariance information is extracted; without the need for overall fine-tuning of large-scale pre-trained models such as CLIP, the recognition robustness of CLIP in dealing with different image content transformations can be enhanced, while the generalization ability of CLIP itself will not be changed.

[0162] In one embodiment, determining a value of a consistency constraint loss function and a value of a classification loss function based on the first similarity vector and the second similarity vector includes:

[0163] Determining a value of a consistency constraint loss function based on the first similarity vector and the second similarity vector, where the value of the consistency constraint loss function is used to characterize a degree of distribution difference between the first similarity vector and the second similarity vector;

[0164] Based on the first similarity vector, a value of a first classification loss function is determined, and based on the second similarity vector, a value of a second classification loss function is determined.

[0165] Specifically, the consistency constraint loss function is used to minimize the distribution difference between the first similarity vector and the second similarity vector, thereby mining the consistency information between the source domain and the target domain (such as cross-domain invariant information). That is, the feature expression of the target domain should be as close as possible to the feature expression of the source domain. Two cross entropy loss functions (the first classification loss function and the second classification loss function) are used to constrain the classification results of the similarity vector of the source domain (the first similarity vector) and the classification results of the similarity vector of the target domain (the second similarity vector).

[0166] In one embodiment, based on the values ​​of the consistency constraint loss function and the classification loss function satisfying predetermined conditions, the domain sharing hint to be trained is updated to obtain the trained domain sharing hint, including:

[0167] If the sum of the values ​​of the consistency constraint loss function, the first classification loss function, and the second classification loss function is equal to or less than the preset loss threshold, the domain sharing prompt to be trained is updated based on the values ​​of the consistency constraint loss function, the first classification loss function, and the second classification loss function to obtain the trained domain sharing prompt.

[0168] Specifically, for the training of the domain-shared hint to be trained, the value of the total loss function = the value of the consistency constraint loss function + the value of the first classification loss function + the value of the second classification loss function.

[0169] If the value of the total loss function is greater than the preset loss threshold, that is, the value of the consistency constraint loss function and the value of the classification loss function do not meet the predetermined conditions, the domain sharing prompt to be trained is updated and steps D1-D6 are repeated; if the value of the total loss function is equal to or less than the preset loss threshold, that is, the value of the consistency constraint loss function and the value of the classification loss function meet the predetermined conditions, the training of the domain sharing prompt to be trained is stopped, and the domain sharing prompt to be trained is updated to obtain the trained domain sharing prompt.

[0170] In one embodiment, the training method provided by the present disclosure includes the following steps:

[0171] Obtain source image samples in the source domain and target image samples in the target domain. The source domain is used to represent the source image sample set, and the target domain is used to represent the target image sample set. The target image samples are obtained by performing image content transformation on the source image samples.

[0172] Determining the value of the consistency constraint loss function and the value of the classification loss function based on the source image sample, the target image sample, the preset candidate category text set, and the obtained domain shared hint to be trained;

[0173] Based on the values ​​of the consistency constraint loss function and the classification loss function, the domain sharing prompt to be trained is updated until the values ​​of the consistency constraint loss function and the classification loss function meet predetermined conditions, and the domain sharing prompt to be trained is updated to obtain the trained domain sharing prompt.

[0174] In one embodiment, based on the source image sample, the target image sample, the preset candidate category text set, and the obtained domain-shared hint to be trained, determining the value of the consistency constraint loss function and the value of the classification loss function includes:

[0175] Determining an image feature vector of the source image sample based on the source image sample and the obtained domain-shared cue to be trained, and determining an image feature vector of the target image sample based on the target image sample and the domain-shared cue to be trained, wherein each learnable vector in the domain-shared cue to be trained is used to represent a vector that can be gradient updated during the training process of the domain-shared cue to be trained;

[0176] Determining a first similarity between the image feature vector of the source image sample and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the source image sample and the text feature vector of each candidate category text, and determining a first similarity vector based on each first similarity, the first similarity vector being used to represent the set of each first similarity;

[0177] Determining a second similarity between the image feature vector of the target image sample and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the target image sample and the text feature vector of each candidate category text, and determining a second similarity vector based on each second similarity, the second similarity vector being used to represent the set of each second similarity;

[0178] Based on the first similarity vector and the second similarity vector, a value of a consistency constraint loss function and a value of a classification loss function are determined.

[0179] In one embodiment, determining a value of a consistency constraint loss function and a value of a classification loss function based on the first similarity vector and the second similarity vector includes:

[0180] Determining a value of a consistency constraint loss function based on the first similarity vector and the second similarity vector, where the value of the consistency constraint loss function is used to characterize a degree of distribution difference between the first similarity vector and the second similarity vector;

[0181] Based on the first similarity vector, a value of a first classification loss function is determined, and based on the second similarity vector, a value of a second classification loss function is determined.

[0182] The application of the embodiments of the present disclosure has at least the following beneficial effects:

[0183] Based on a fixed CLIP, domain-shared cues are trained by constraining the feature representations of the source and target domains (consistency constraint loss function), making CLIP robust to both source and target domains. Feature alignment between source and target domain images is achieved, and domain-invariant information is extracted. Without requiring overall fine-tuning of large-scale pre-trained models like CLIP, CLIP's robustness to image content transformations is enhanced without changing its inherent generalization capabilities. Based on the preset domain-shared cues (post-training domain-shared cues), cross-domain invariant information is extracted during target image recognition, enhancing recognition robustness for target images (e.g., when the target image is in the target domain) under varying image content transformations and improving recognition accuracy for target images.

[0184] In order to better understand the method provided by the embodiment of the present disclosure, the solution of the embodiment of the present disclosure is further described below with reference to examples of specific application scenarios.

[0185] In one embodiment, the method provided by the disclosed embodiments can be applied to online video recognition. By deploying trained domain-sharing hints, the trained domain-sharing hints are used to extract cross-domain invariant information about images in the video to be recognized, thereby improving recognition accuracy for images in the video to be recognized. The method provided by the disclosed embodiments can also be applied to products such as content review.

[0186] In a specific application scenario embodiment, such as an image recognition scenario, see Figure 5 , shows a processing flow of an image recognition method, such as Figure 5 As shown, the processing flow of the image recognition method provided by the embodiment of the present disclosure includes the following steps:

[0187] S401: The server obtains source image samples in a source domain and target image samples in a target domain.

[0188] Specifically, the source image samples in the source domain are source domain images, and the target image samples in the target domain are target domain images.

[0189] S402: The server determines an image feature vector of the source image sample based on the source image sample and the shared hint of the domain to be trained, and determines an image feature vector of the target image sample based on the shared hint of the target image sample and the domain to be trained.

[0190] Specifically, for example, Figure 4As shown, based on the source domain image (source image sample) and the domain shared prompt (domain shared prompt to be trained), the image feature vector of the source domain image (source image sample) is determined; and based on the target domain image (target image sample) and the domain shared prompt (domain shared prompt to be trained), the image feature vector of the target domain image (target image sample) is determined.

[0191] S403, the server determines a first similarity between the image feature vector of the source image sample and the text feature vector of each candidate category text in the candidate category text set based on the image feature vector of the source image sample and the text feature vector of each candidate category text, and determines a first similarity vector based on each first similarity.

[0192] Specifically, for example, Figure 4 As shown, multiple image categories are obtained; based on the multiple image categories and prompt word encapsulation, candidate category text sets corresponding to the multiple image categories are determined; word embedding vector groups (text tokens) corresponding to the candidate category text sets are determined; based on the word embedding vector groups, text feature vectors for each candidate category text in the candidate category text set are obtained through the text encoder in CLIP. A similarity vector is a set of similarities. For example, a first similarity vector is a set of first similarities.

[0193] S404, the server determines a second similarity between the image feature vector of the target image sample and the text feature vector of each candidate category text in the candidate category text set based on the image feature vector of the target image sample and the text feature vector of each candidate category text, and determines a second similarity vector based on each second similarity, where the second similarity vector is used to represent the set of each second similarity.

[0194] Specifically, the similarity vector is a set of similarities, for example, the second similarity vector is a set of second similarities.

[0195] S405: The server determines a value of a consistency constraint loss function and a value of a classification loss function based on the first similarity vector and the second similarity vector.

[0196] Specifically, for example, Figure 4 As shown, based on the first similarity vector and the second similarity vector, the value of the consistency constraint loss function is determined; based on the first similarity vector, the value of the first classification loss function is determined, and based on the second similarity vector, the value of the second classification loss function is determined.

[0197] S406, the server determines whether the value of the consistency constraint loss function and the value of the classification loss function meet the predetermined conditions. If the value of the consistency constraint loss function and the value of the classification loss function do not meet the predetermined conditions, execute S407; if the value of the consistency constraint loss function and the value of the classification loss function meet the predetermined conditions, execute S408.

[0198] Specifically, if the value of the consistency constraint loss function and the value of the classification loss function do not meet the predetermined conditions, the domain sharing prompt to be trained is updated and steps S401-S407 are repeated; if the value of the consistency constraint loss function and the value of the classification loss function meet the predetermined conditions, the training of the domain sharing prompt to be trained is stopped, and the domain sharing prompt to be trained is updated to obtain the trained domain sharing prompt.

[0199] S407: The server updates the domain sharing prompt to be trained based on the value of the consistency constraint loss function and the value of the classification loss function, and goes to S401 for execution.

[0200] Specifically, for example, for the training of the domain-sharing prompt to be trained, the value of the total loss function = the value of the consistency constraint loss function + the value of the first classification loss function + the value of the second classification loss function; if the value of the total loss function is greater than the preset loss threshold, that is, the value of the consistency constraint loss function and the value of the classification loss function do not meet the predetermined conditions, the domain-sharing prompt to be trained is updated.

[0201] S408 : The server updates the domain sharing prompt to be trained based on the value of the consistency constraint loss function and the value of the classification loss function to obtain the trained domain sharing prompt.

[0202] Specifically, for example, for the training of the domain shared prompt to be trained, the value of the total loss function = the value of the consistency constraint loss function + the value of the first classification loss function + the value of the second classification loss function; if the value of the total loss function is equal to or less than the preset loss threshold, that is, the value of the consistency constraint loss function and the value of the classification loss function meet the predetermined conditions, then the training of the domain shared prompt to be trained is stopped, and the domain shared prompt to be trained is updated to obtain the trained domain shared prompt.

[0203] S409 : The server determines an image feature vector of the target image based on the acquired target image and the trained domain sharing hint.

[0204] Specifically, a target image, such as a source domain image or a target domain image, is obtained and preprocessed to determine a preprocessed target image; based on the preprocessed target image, the preprocessed target image is segmented into a plurality of image blocks; each of the plurality of image blocks is mapped to a low-dimensional vector, and each low-dimensional vector is constructed into an image tag sequence corresponding to the plurality of image blocks; and based on the image tag sequence corresponding to the plurality of image blocks and a plurality of learnable vectors in a preset domain-shared prompt, an image feature vector of the target image is determined.

[0205] S410 , the server determines the similarity between the image feature vector of the target image and the text feature vector of each candidate category text in the candidate category text set based on the image feature vector of the target image and the text feature vector of each candidate category text in the candidate category text set.

[0206] Specifically, for example, the candidate category text set includes 1000 candidate category texts, and the similarities between the image feature vector of the target image and the text feature vectors of the 1000 candidate category texts, that is, 1000 similarities, are determined.

[0207] S411: The server determines the image category of the target image based on the similarities.

[0208] Specifically, for example, the candidate category text set includes 1,000 candidate category texts, and the image category of the target image is determined based on the similarity between the image feature vector of the target image and the text feature vectors of the 1,000 candidate category texts, that is, based on the 1,000 similarities; for example, the image category described in the candidate category text corresponding to the maximum similarity among these 1,000 similarities is determined as the image category of the target image; for example, if the image category described in the candidate category text corresponding to the maximum similarity is dog, then the image category of the target image is determined to be dog.

[0209] The application of the embodiments of the present disclosure has at least the following beneficial effects:

[0210] Based on a fixed CLIP, domain-shared cues are trained by applying a consistency constraint (consistency constraint loss function) on the feature representations of the source and target domains, making CLIP robust to both source and target domains. Feature alignment between source and target images is achieved, and domain-invariant information is extracted. The domain-shared cues are shared by the input images of the source and target domains, allowing for the sharing of input information between the two domains (source and target) during training, facilitating the mining of cross-domain invariant information. Without requiring fine-tuning of large-scale pre-trained models like CLIP, CLIP's robustness to image content transformations is enhanced without compromising its inherent generalization capabilities. Based on the preset domain-shared cues (trained domain-shared cues), cross-domain invariant information is extracted during target image recognition, enhancing robustness to target images (e.g., target images in the target domain) under varying image content transformations and improving recognition accuracy for target images.

[0211] The present disclosure also provides an image recognition device. The structural diagram of the image recognition device is shown in FIG. Figure 6 As shown, the image recognition device 50 includes a first processing module 501 , a second processing module 502 and a third processing module 503 .

[0212] A first processing module 501 is configured to determine an image feature vector of a target image based on an acquired target image and a preset domain sharing hint, wherein the preset domain sharing hint is used to represent cross-domain invariant information, and the cross-domain invariant information includes image information other than image appearance changes;

[0213] A second processing module 502 is configured to determine a similarity between the image feature vector of the target image and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the target image and the text feature vector of each candidate category text, wherein each candidate category text is used to describe an image category;

[0214] The third processing module 503 is configured to determine the image category of the target image based on the similarities.

[0215] In one embodiment, the first processing module 501 is specifically configured to:

[0216] Determine a preprocessed target image based on the acquired target image through preprocessing;

[0217] Based on the preprocessed target image, the preprocessed target image is segmented into a plurality of image blocks through segmentation processing;

[0218] Mapping each image block in the plurality of image blocks into a low-dimensional vector, and constructing each low-dimensional vector into an image label sequence corresponding to the plurality of image blocks;

[0219] An image feature vector of a target image is determined based on an image tag sequence corresponding to a plurality of image blocks and a plurality of learnable vectors in a preset domain-sharing hint, wherein each of the plurality of learnable vectors is used to represent a vector capable of gradient updating.

[0220] In one embodiment, the first processing module 501 is specifically configured to:

[0221] Determine a cascaded vector based on image tag sequences corresponding to the plurality of image blocks and a plurality of learnable vectors in a preset domain-shared hint through cascade processing;

[0222] Based on the vector after cascade processing, the image feature vector of the target image is obtained through the image encoder in the preset pre-training model.

[0223] In one embodiment, the second processing module 502 is further configured to:

[0224] Get multiple image categories;

[0225] Based on the multiple image categories and the preset prompt word encapsulation, determining candidate category text sets corresponding to the multiple image categories, and determining the candidate category text sets corresponding to the multiple image categories as the preset candidate category text set;

[0226] Determine the word embedding vector group corresponding to the preset candidate category text set;

[0227] Based on the word embedding vector group, the text feature vector of each candidate category text in the preset candidate category text set is obtained through the text encoder in the preset pre-trained model.

[0228] In one embodiment, the third processing module 503 is specifically configured to:

[0229] The maximum similarity among the similarities is determined, and the image category described in the candidate category text corresponding to the maximum similarity is determined as the image category of the target image.

[0230] In one embodiment, the first processing module 501 is further configured to:

[0231] Obtain source image samples in the source domain and target image samples in the target domain. The source domain is used to represent the source image sample set, and the target domain is used to represent the target image sample set. The target image samples are obtained by performing image content transformation on the source image samples.

[0232] Determining an image feature vector of the source image sample based on the source image sample and the obtained domain-shared cue to be trained, and determining an image feature vector of the target image sample based on the target image sample and the domain-shared cue to be trained, wherein each learnable vector in the domain-shared cue to be trained is used to represent a vector that can be gradient updated during the training process of the domain-shared cue to be trained;

[0233] Determining a first similarity between the image feature vector of the source image sample and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the source image sample and the text feature vector of each candidate category text, and determining a first similarity vector based on each first similarity, the first similarity vector being used to represent the set of each first similarity;

[0234] Determining a second similarity between the image feature vector of the target image sample and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the target image sample and the text feature vector of each candidate category text, and determining a second similarity vector based on each second similarity, the second similarity vector being used to represent the set of each second similarity;

[0235] Determining a value of a consistency constraint loss function and a value of a classification loss function based on the first similarity vector and the second similarity vector;

[0236] Based on the values ​​of the consistency constraint loss function and the classification loss function, the domain sharing prompt to be trained is updated until the values ​​of the consistency constraint loss function and the classification loss function meet predetermined conditions, and the domain sharing prompt to be trained is updated to obtain the trained domain sharing prompt.

[0237] In one embodiment, the first processing module 501 is specifically configured to:

[0238] Determining a value of a consistency constraint loss function based on the first similarity vector and the second similarity vector, where the value of the consistency constraint loss function is used to characterize a degree of distribution difference between the first similarity vector and the second similarity vector;

[0239] Based on the first similarity vector, a value of a first classification loss function is determined, and based on the second similarity vector, a value of a second classification loss function is determined.

[0240] In one embodiment, the first processing module 501 is specifically configured to:

[0241] If the sum of the values ​​of the consistency constraint loss function, the first classification loss function, and the second classification loss function is equal to or less than the preset loss threshold, the domain sharing prompt to be trained is updated based on the values ​​of the consistency constraint loss function, the first classification loss function, and the second classification loss function to obtain the trained domain sharing prompt.

[0242] The application of the embodiments of the present disclosure has at least the following beneficial effects:

[0243] Based on the acquired target image and the preset domain sharing hint, the image feature vector of the target image is determined. The preset domain sharing hint is used to represent cross-domain invariant information, which includes image information that does not change in image appearance. Based on the image feature vector of the target image and the text feature vector of each candidate category text in the preset candidate category text set, the similarity between the image feature vector of the target image and the text feature vector of each candidate category text is determined. Each candidate category text is used to describe an image category. Based on each similarity, the image category of the target image is determined. In this way, based on the preset domain sharing hint (the trained domain sharing hint), the cross-domain invariant information of the target image is extracted during the target image recognition process, the recognition robustness of the target image (for example, the target image is obtained through image content transformation) is enhanced when dealing with different image content transformations, and the accuracy of image recognition for the target image is improved.

[0244] The embodiment of the present disclosure further provides a training device, the training device 60 , including a fourth processing module 601 .

[0245] A fourth processing module 601 is configured to obtain source image samples in a source domain and target image samples in a target domain, wherein the source domain is used to represent a set of source image samples, and the target domain is used to represent a set of target image samples, and the target image samples are obtained by performing image content transformation on the source image samples;

[0246] Determining the value of the consistency constraint loss function and the value of the classification loss function based on the source image sample, the target image sample, the preset candidate category text set, and the obtained domain shared hint to be trained;

[0247] Based on the values ​​of the consistency constraint loss function and the classification loss function, the domain sharing prompt to be trained is updated until the values ​​of the consistency constraint loss function and the classification loss function meet predetermined conditions, and the domain sharing prompt to be trained is updated to obtain the trained domain sharing prompt.

[0248] In one embodiment, the fourth processing module 601 is specifically configured to:

[0249] Determining an image feature vector of the source image sample based on the source image sample and the obtained domain-shared cue to be trained, and determining an image feature vector of the target image sample based on the target image sample and the domain-shared cue to be trained, wherein each learnable vector in the domain-shared cue to be trained is used to represent a vector that can be gradient updated during the training process of the domain-shared cue to be trained;

[0250] Determining a first similarity between the image feature vector of the source image sample and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the source image sample and the text feature vector of each candidate category text, and determining a first similarity vector based on each first similarity, the first similarity vector being used to represent the set of each first similarity;

[0251] Determining a second similarity between the image feature vector of the target image sample and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the target image sample and the text feature vector of each candidate category text, and determining a second similarity vector based on each second similarity, the second similarity vector being used to represent the set of each second similarity;

[0252] Based on the first similarity vector and the second similarity vector, a value of a consistency constraint loss function and a value of a classification loss function are determined.

[0253] In one embodiment, the fourth processing module 601 is specifically configured to:

[0254] Determining a value of a consistency constraint loss function based on the first similarity vector and the second similarity vector, where the value of the consistency constraint loss function is used to characterize a degree of distribution difference between the first similarity vector and the second similarity vector;

[0255] Based on the first similarity vector, a value of a first classification loss function is determined, and based on the second similarity vector, a value of a second classification loss function is determined.

[0256] The present disclosure also provides an electronic device. The structural diagram of the electronic device is as follows: Figure 7 As shown, Figure 7 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present disclosure.

[0257] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0258] Bus 4002 may include a path for transmitting information between the above components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0259] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0260] The memory 4003 is used to store the computer program for executing the embodiments of the present disclosure, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiments.

[0261] Among them, electronic equipment includes but is not limited to: servers, etc.

[0262] The application of the embodiments of the present disclosure has at least the following beneficial effects:

[0263] Based on the acquired target image and the preset domain sharing hint, the image feature vector of the target image is determined. The preset domain sharing hint is used to represent cross-domain invariant information, which includes image information that does not change in image appearance. Based on the image feature vector of the target image and the text feature vector of each candidate category text in the preset candidate category text set, the similarity between the image feature vector of the target image and the text feature vector of each candidate category text is determined. Each candidate category text is used to describe an image category. Based on each similarity, the image category of the target image is determined. In this way, based on the preset domain sharing hint (the trained domain sharing hint), the cross-domain invariant information of the target image is extracted during the target image recognition process, the recognition robustness of the target image (for example, the target image is obtained through image content transformation) is enhanced when dealing with different image content transformations, and the accuracy of image recognition for the target image is improved.

[0264] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0265] The embodiments of the present disclosure further provide a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiments when executed by a processor.

[0266] It should be understood that, although the flowcharts of the embodiments of the present disclosure indicate the various operation steps by arrows, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiments of the present disclosure, the implementation steps in each flowchart can be performed in other orders as required. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times. In scenarios where the execution times are different, the order of execution of these sub-steps or stages can be flexibly configured as required, and the embodiments of the present disclosure do not limit this.

[0267] The above description is only an optional implementation method for some implementation scenarios of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present disclosure, other similar implementation methods based on the technical ideas of the present disclosure also fall within the protection scope of the embodiments of the present disclosure.

Claims

1. An image recognition method, characterized in that: include: Determining an image feature vector of the target image based on the acquired target image and a preset domain sharing hint, wherein the preset domain sharing hint is used to represent cross-domain invariant information, wherein the cross-domain invariant information includes image information other than image appearance changes; Determining a similarity between the image feature vector of the target image and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the target image and the text feature vector of each candidate category text, wherein each candidate category text is used to describe an image category; Based on the similarities, an image category of the target image is determined.

2. The method according to claim 1, characterized in that The determining, based on the acquired target image and the preset domain sharing prompt, an image feature vector of the target image includes: Determine a preprocessed target image based on the acquired target image through preprocessing; Based on the preprocessed target image, dividing the preprocessed target image into a plurality of image blocks through segmentation processing; Mapping each image block in the plurality of image blocks into a low-dimensional vector, and constructing each low-dimensional vector into an image label sequence corresponding to the plurality of image blocks; An image feature vector of the target image is determined based on image label sequences corresponding to the multiple image blocks and multiple learnable vectors in the preset domain-sharing prompt, each learnable vector in the multiple learnable vectors being used to represent a vector capable of gradient updating.

3. The method according to claim 2, characterized in that The determining of the image feature vector of the target image based on the image tag sequences corresponding to the plurality of image blocks and the plurality of learnable vectors in the preset domain-shared hints includes: Determining a cascaded vector based on the image tag sequences corresponding to the plurality of image blocks and the plurality of learnable vectors in the preset domain-sharing hint by cascading processing; Based on the vector after the cascade processing, the image feature vector of the target image is obtained through the image encoder in the preset pre-training model.

4. The method according to claim 1, wherein Before determining the similarity between the image feature vector of the target image and the text feature vector of each candidate category text in the preset candidate category text set based on the image feature vector of the target image and the text feature vector of each candidate category text, the method further includes: Get multiple image categories; Determining candidate category text sets corresponding to the multiple image categories based on the multiple image categories and the preset prompt word encapsulation, and determining the candidate category text sets corresponding to the multiple image categories as the preset candidate category text set; Determine a word embedding vector group corresponding to the preset candidate category text set; Based on the word embedding vector group, a text feature vector of each candidate category text in the preset candidate category text set is obtained through a text encoder in a preset pre-training model.

5. The method according to claim 1, characterized in that The determining the image category of the target image based on the similarities includes: The maximum similarity among the similarities is determined, and the image category described in the candidate category text corresponding to the maximum similarity is determined as the image category of the target image.

6. The method according to claim 1, characterized in that Before determining the image feature vector of the target image based on the acquired target image and the preset domain sharing prompt, the method further includes: Obtaining source image samples in a source domain and target image samples in a target domain, wherein the source domain is used to represent a set of source image samples, the target domain is used to represent a set of target image samples, and the target image samples are obtained by performing image content transformation on the source image samples; Determining an image feature vector of the source image sample based on the source image sample and the obtained domain-shared cue to be trained, and determining an image feature vector of the target image sample based on the target image sample and the domain-shared cue to be trained, wherein each learnable vector in the domain-shared cue to be trained is used to represent a vector that can be gradient-updated during the training process of the domain-shared cue to be trained; Determining, based on the image feature vector of the source image sample and the text feature vector of each candidate category text in the preset candidate category text set, a first similarity between the image feature vector of the source image sample and the text feature vector of each candidate category text, and determining, based on each first similarity, a first similarity vector, the first similarity vector being used to represent the set of each first similarity; Determining, based on the image feature vector of the target image sample and the text feature vector of each candidate category text in the preset candidate category text set, a second similarity between the image feature vector of the target image sample and the text feature vector of each candidate category text, and determining, based on each second similarity, a second similarity vector, the second similarity vector being used to represent the set of each second similarity; Determining a value of a consistency constraint loss function and a value of a classification loss function based on the first similarity vector and the second similarity vector; Based on the value of the consistency constraint loss function and the value of the classification loss function, the domain sharing prompt to be trained is updated until the value of the consistency constraint loss function and the value of the classification loss function meet a predetermined condition, and the domain sharing prompt to be trained is updated to obtain a trained domain sharing prompt.

7. The method according to claim 6, characterized in that The determining, based on the first similarity vector and the second similarity vector, a value of a consistency constraint loss function and a value of a classification loss function includes: Determining a value of a consistency constraint loss function based on the first similarity vector and the second similarity vector, where the value of the consistency constraint loss function is used to characterize a degree of distribution difference between the first similarity vector and the second similarity vector; Based on the first similarity vector, a value of a first classification loss function is determined, and based on the second similarity vector, a value of a second classification loss function is determined.

8. The method according to claim 7, characterized in that The updating of the domain sharing hint to be trained based on the fact that the value of the consistency constraint loss function and the value of the classification loss function meet a predetermined condition to obtain the trained domain sharing hint includes: If the sum of the value of the consistency constraint loss function, the value of the first classification loss function and the value of the second classification loss function is equal to or less than a preset loss threshold, the domain sharing prompt to be trained is updated based on the value of the consistency constraint loss function, the value of the first classification loss function and the value of the second classification loss function to obtain the trained domain sharing prompt.

9. A training method, characterized in that: include: Obtaining source image samples in a source domain and target image samples in a target domain, wherein the source domain is used to represent a set of source image samples, the target domain is used to represent a set of target image samples, and the target image samples are obtained by performing image content transformation on the source image samples; Determining a value of a consistency constraint loss function and a value of a classification loss function based on the source image sample, the target image sample, a preset candidate category text set, and the obtained domain-shared hint to be trained; Based on the value of the consistency constraint loss function and the value of the classification loss function, the domain sharing prompt to be trained is updated until the value of the consistency constraint loss function and the value of the classification loss function meet a predetermined condition, and the domain sharing prompt to be trained is updated to obtain a trained domain sharing prompt.

10. The method according to claim 9, characterized in that The determining of a value of a consistency constraint loss function and a value of a classification loss function based on the source image sample, the target image sample, a preset candidate category text set, and the obtained domain sharing hint to be trained includes: Determining an image feature vector of the source image sample based on the source image sample and the obtained domain-shared cue to be trained, and determining an image feature vector of the target image sample based on the target image sample and the domain-shared cue to be trained, wherein each learnable vector in the domain-shared cue to be trained is used to represent a vector that can be gradient-updated during the training process of the domain-shared cue to be trained; Determining, based on the image feature vector of the source image sample and the text feature vector of each candidate category text in the preset candidate category text set, a first similarity between the image feature vector of the source image sample and the text feature vector of each candidate category text, and determining, based on each first similarity, a first similarity vector, the first similarity vector being used to represent the set of each first similarity; Determining, based on the image feature vector of the target image sample and the text feature vector of each candidate category text in the preset candidate category text set, a second similarity between the image feature vector of the target image sample and the text feature vector of each candidate category text, and determining, based on each second similarity, a second similarity vector, the second similarity vector being used to represent the set of each second similarity; Based on the first similarity vector and the second similarity vector, a value of a consistency constraint loss function and a value of a classification loss function are determined.

11. The method according to claim 10, characterized in that The determining, based on the first similarity vector and the second similarity vector, a value of a consistency constraint loss function and a value of a classification loss function includes: Determining a value of a consistency constraint loss function based on the first similarity vector and the second similarity vector, where the value of the consistency constraint loss function is used to characterize a degree of distribution difference between the first similarity vector and the second similarity vector; Based on the first similarity vector, a value of a first classification loss function is determined, and based on the second similarity vector, a value of a second classification loss function is determined.

12. An image recognition device, characterized in that: include: a first processing module, configured to determine an image feature vector of the target image based on the acquired target image and a preset domain sharing hint, wherein the preset domain sharing hint is used to represent cross-domain invariant information, wherein the cross-domain invariant information includes image information other than image appearance changes; a second processing module, configured to determine a similarity between the image feature vector of the target image and the text feature vector of each candidate category text in a preset candidate category text set based on the image feature vector of the target image and the text feature vector of each candidate category text, wherein each candidate category text is used to describe an image category; The third processing module is configured to determine the image category of the target image based on the similarities.

13. A training device, characterized in that: include: a fourth processing module, configured to obtain source image samples in a source domain and target image samples in a target domain, wherein the source domain is used to represent a set of source image samples, the target domain is used to represent a set of target image samples, and the target image samples are obtained by performing image content transformation on the source image samples; Determining a value of a consistency constraint loss function and a value of a classification loss function based on the source image sample, the target image sample, a preset candidate category text set, and the obtained domain-shared hint to be trained; Based on the value of the consistency constraint loss function and the value of the classification loss function, the domain sharing prompt to be trained is updated until the value of the consistency constraint loss function and the value of the classification loss function meet a predetermined condition, and the domain sharing prompt to be trained is updated to obtain a trained domain sharing prompt.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 11.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images