Method and system for image recognition

Abstract

The method and system for image recognition provided in the specification obtain a training sample set, extract at least one training sample pair from the training sample set, perform feature extraction on each training sample pair by using a recognition model to obtain image features of the image sample and text features of the text sample, then extract at least one image sub-feature corresponding to a feature region of a preset size from the image features, and identify text sub-features corresponding to each text word in the text sample from the text features, traverse the similarity between the at least one image sub-feature and the text sub-feature to obtain a local similarity set, and converge the recognition model based on the local similarity set to obtain a target recognition model, and identify attribute information of a target object in an object image by using the target recognition model; the scheme can improve the recognition accuracy of image recognition by refining the constraint granularity of the recognition model.

Description

Technical Field

[0001] This specification relates to the field of image recognition, and in particular to an image recognition method and system. Background Technology

[0002] In recent years, with the rapid development of internet technology, object profiling information has become increasingly important. Profiling information can be understood as the attribute information of an object; for example, if the object is a merchant, it would be the merchant's attribute information. Currently, profiling information is mainly collected through text and statistical information, often neglecting the profiling information contained in images. Therefore, it is possible to identify profiling information within images. Existing image recognition methods often obtain profiling information through image classification.

[0003] In the process of researching and practicing existing technologies, the inventors of this invention discovered that image classification requires matching image features with text features of preset attribute information through a recognition model. In the training of the recognition model, the relationship between the image and the text is used for coarse-grained constraints, which reduces the accuracy of matching. Therefore, the accuracy of image recognition is low.

[0004] Therefore, there is a need for a method and system for image recognition with higher accuracy. Summary of the Invention

[0005] This specification provides a method and system for image recognition with higher accuracy.

[0006] In a first aspect, this specification provides an image recognition method, comprising: acquiring a training sample set, and extracting at least one training sample pair from the training sample set, each training sample pair including an image sample and a text sample; using a recognition model to extract features from each training sample pair to obtain image features of the image sample and text features of the text sample; extracting at least one image sub-feature corresponding to a feature region of a preset size from the image features, and identifying text sub-features corresponding to each text word in the text sample from the text features; traversing the similarity between the at least one image sub-feature and the text sub-feature to obtain a local similarity set, and converging the recognition model based on the local similarity set to obtain a target recognition model; and using the target recognition model to identify attribute information of a target object in an object image.

[0007] In some embodiments, the training sample set includes a full sample set of all objects and a test sample set of the target object, wherein the number of samples in the full sample set is greater than the number of samples in the test sample set; and the step of extracting at least one training sample pair from the training sample set includes: extracting at least one candidate sample pair from the full sample set; extracting at least one test sample pair from the test sample set; and using the at least one candidate sample pair and the at least one test sample pair as the at least one training sample pair.

[0008] In some embodiments, the step of extracting at least one image sub-feature corresponding to a feature region of a preset size from the image features and identifying the text sub-feature corresponding to each text word in the text sample from the text features includes: obtaining the similarity between the image features and the text features to obtain a global similarity set corresponding to the at least one training sample pair; selecting a target training sample pair from the at least one training sample pair based on the global similarity set; and extracting at least one image sub-feature corresponding to a feature region of a preset size from the image features based on the target training sample pair and identifying the text sub-feature corresponding to each text word in the text sample from the text features.

[0009] In some embodiments, the target training sample pair includes target image samples and target text samples; and the step of extracting at least one image sub-feature corresponding to a feature region of a preset size from the image features based on the target training sample pair, and identifying the text sub-feature corresponding to each text word in the text sample from the text features, includes: filtering target image features of the target image sample from the image features, and filtering target text features of the target text sample from the text features; extracting at least one image sub-feature corresponding to a feature region of a preset size from the target image features; and extracting the text sub-feature corresponding to each text word in the target text sample from the target text features.

[0010] In some embodiments, selecting target training sample pairs from the at least one training sample pair based on the global similarity set includes: determining global loss information corresponding to each training sample pair based on the global similarity set; determining the sampling probability of each training sample pair based on the image features, the text features, and the global loss information corresponding to each training sample pair; and selecting target training sample pairs from the at least one training sample pair based on the sampling probability.

[0011] In some embodiments, determining the global loss information corresponding to each training sample pair based on the global similarity set includes: filtering out the similarity corresponding to each training sample pair from the global similarity set to obtain a target global similarity; obtaining the feature difference between the image feature and the text feature corresponding to each training sample pair based on the target global similarity; and determining the global loss information corresponding to each training sample pair based on the feature difference.

[0012] In some embodiments, determining the sampling probability of each training sample pair based on the image features, text features, and global loss information corresponding to each training sample pair includes: determining the gradient corresponding to each training sample pair based on the global loss information; and determining the sampling probability of each training sample pair based on the image features, text features, and gradient corresponding to each training sample pair through the sampling network of the recognition model.

[0013] In some embodiments, the recognition model includes an encoding network and a sampling network; and the convergence of the recognition model based on the local similarity set to obtain a target recognition model includes: performing a preset first number of encoding training operations on the encoding network based on the sampling network, the global loss information, and the local similarity set to obtain a target encoding network; performing a preset second number of sampling training operations on the sampling network based on the target encoding network to obtain a target sampling network; using the target encoding network and the target sampling network as the current recognition model, and updating the recognition model to the current recognition model; and returning to the step of using the recognition model to extract features from each training sample pair until the recognition model converges to obtain the target recognition model.

[0014] In some embodiments, the preset first number is greater than the second preset number, and the ratio between the preset first number and the preset second number is a preset ratio.

[0015] In some embodiments, the step of performing a preset number of encoding training iterations on the encoding network based on the sampling network, the global loss information, and the local similarity set to obtain a target encoding network includes: determining the encoding loss information of the encoding network based on the global loss information and the local similarity set; updating the network parameters of the encoding network based on the encoding loss information to obtain a current encoding network; using the current encoding network and the sampling network as candidate recognition models, and updating the recognition model to the candidate recognition model; and returning to the step of using the recognition model to extract features from each training sample pair until the preset number of updates is reached to obtain the target encoding network.

[0016] In some embodiments, determining the encoding loss information of the encoding network based on the global loss information and the local similarity set includes: selecting the target global loss information corresponding to the target training sample pair from the global loss information; determining the local loss information corresponding to the encoding network based on the local similarity set; and fusing the global loss information and the local loss information to obtain the encoding loss information of the encoding network.

[0017] In some embodiments, determining the local loss information corresponding to the encoding network based on the local similarity set includes: traversing the local similarity corresponding to each text sub-feature in the local similarity set to obtain a candidate local similarity set corresponding to each text sub-feature; selecting the local similarity with the highest similarity in the candidate local similarity set to obtain the target local similarity corresponding to each text sub-feature; and adjusting the target local similarity to a preset similarity to obtain the local loss information corresponding to the encoding network, wherein the preset similarity is greater than the target local similarity.

[0018] In some embodiments, the step of performing a preset second sampling training on the sampling network based on the target encoding network to obtain the target sampling network includes: using the target encoding network to extract features from the at least one training sample pair to obtain a current global loss information set corresponding to the at least one training sample pair; filtering candidate global loss information from the current global loss information set based on the sampling network, and extracting target test loss information from the candidate global loss information; updating the network parameters of the sampling network based on the target test loss information to obtain the current sampling network, and updating the sampling network to the current sampling network; and returning to the step of using the target encoding network to extract features from the at least one training sample pair until the number of updates reaches the preset second number, thereby obtaining the target sampling network.

[0019] In some embodiments, extracting the target test loss information from the candidate global loss information includes: selecting at least one global loss information corresponding to a test sample pair from the candidate global loss information to obtain a test loss information set; and selecting the minimum test loss information from the test loss information set to obtain the target test loss information.

[0020] In some embodiments, before extracting features from each training sample pair using the recognition model, the method further includes: using a sampling network in a preset recognition model and the at least one training sample pair to perform a preset first number of encoding training on the encoding network in the preset recognition model to obtain a basic encoding network; based on the basic encoding network and the at least one training sample pair, performing a preset second number of sampling training on the sampling network in the preset recognition model to obtain a basic sampling network; using the basic encoding network and the basic sampling network as a basic recognition model, and updating the preset recognition model to the basic recognition model; and returning to the step of performing a preset first number of encoding training on the encoding network in the preset recognition model using the sampling network in the preset recognition model and the at least one training sample pair, until the preset recognition model converges to obtain the recognition model.

[0021] In some embodiments, the step of using the sampling network in the preset recognition model and the at least one training sample pair to perform a preset first-time encoding training on the encoding network in the preset recognition model to obtain a basic encoding network includes: using the encoding network in the preset recognition model to extract features from the at least one training sample pair to obtain a candidate global loss information set corresponding to the at least one training sample pair; selecting basic global loss information from the candidate global loss information set based on the sampling network in the preset recognition model; and performing the preset first-time encoding training on the encoding network in the preset recognition model based on the basic global loss information to obtain the basic encoding network.

[0022] In some embodiments, after the recognition model is converged based on the local similarity set to obtain the target recognition model, the method further includes: obtaining a text set corresponding to at least one preset attribute information of the target object; and using the target recognition model to extract features from each text in the text set to obtain a text feature set corresponding to the at least one preset attribute information.

[0023] In some embodiments, the step of identifying the attribute information of a target object in an object image using the target recognition model includes: acquiring an object image of the target object; extracting features from the object image using the target recognition model to obtain current image features of the object image; and determining the attribute information of the target object in the object image based on the text feature set and the current image features.

[0024] In some embodiments, determining the attribute information of the target object in the object image based on the current image features of the text feature set includes: obtaining the similarity between the current image features and each text feature in the text feature set to obtain a current similarity set; filtering out target similarities exceeding a preset similarity threshold from the current similarity set; and filtering out attribute information corresponding to the target similarity from at least one attribute information to obtain the attribute information of the target object.

[0025] In some embodiments, the method further includes: obtaining the target text corresponding to the current attribute information, and updating the at least one attribute information based on the current attribute information to obtain an updated attribute information set; using the target recognition model to extract features from the target text to obtain current text features; and updating the text feature set based on the current text features to obtain an updated text feature set corresponding to the attribute information set.

[0026] Secondly, this specification also provides an image recognition system, comprising: at least one storage medium storing at least one instruction set for performing image recognition; and at least one processor communicatively connected to the at least one storage medium, wherein, when the image recognition system is running, the at least one processor reads the at least one instruction set and executes the image recognition method described in the first aspect of this specification according to the instructions of the at least one instruction set.

[0027] As can be seen from the above technical solutions, the image recognition method and system provided in this specification obtain a training sample set, and after extracting at least one training sample pair from the training sample set, each training sample pair includes an image sample and a text sample. A recognition model is used to extract features from each training sample pair to obtain image features of the image sample and text features of the text sample. Then, at least one image sub-feature corresponding to a feature region of a preset size is extracted from the image features, and the text sub-feature corresponding to each text word in the text sample is identified from the text features. The process then iterates through the at least one image sub-feature and the text sub-feature. Similarity is used to obtain a set of local similarities, and the recognition model is converged based on the set of local similarities to obtain a target recognition model. The target recognition model is then used to identify the attribute information of the target object in the object image. Since this scheme extracts image sub-features from image features and extracts text sub-features of each text word in the text sample during the training process of the recognition model, and then iterates through the similarity between image sub-features and text sub-features to constrain them, the granularity of the constraint is refined to the relationship between feature regions and text words, thereby improving the recognition accuracy of the recognition model. Therefore, the recognition accuracy of image recognition can be improved.

[0028] Other functions of the image recognition methods and systems provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the image recognition methods and systems provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A schematic diagram illustrating an application scenario of an image recognition system provided according to an embodiment of this specification is shown.

[0031] Figure 2 A hardware structure diagram of a computing device provided according to an embodiment of this specification is shown;

[0032] Figure 3 A flowchart of an image recognition method according to embodiments of this specification is shown; and

[0033] Figure 4 A schematic diagram of a process for obtaining a merchant profile is shown according to an embodiment of this specification. Detailed Implementation

[0034] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0035] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0036] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0037] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0038] Before describing the specific embodiments in this specification, the application scenarios of this specification will be introduced as follows:

[0039] In merchant profiling scenarios, profiling technology provides a solid data foundation for various marketing and promotional activities. The more granular the information profiling technology provides, the more relevant the tools and resources operations and event planning can utilize. Existing profiling technologies obtain various profile attributes from information uploaded by merchants and interactions on apps, often neglecting the diverse merchant profile information contained within images. When recognizing merchant profile information in images, image classification is often used, directly matching the image features of the merchant image with the text features corresponding to preset attribute information through a recognition model. In the process of feature matching across multiple models, the recognition model typically employs coarse-grained constraints between image and text features, resulting in low accuracy in matching merchant profiles. If more granular constraints could be used during the training phase of the recognition model, the recognition accuracy could be improved, thereby increasing the accuracy of identifying merchant profiles from object images.

[0040] For ease of description, the terms that will appear in the following descriptions will be explained as follows:

[0041] Multimodal relationships: These usually refer to the relationships between objects of different modalities. In this scheme, they specifically refer to the similarity between images and text. For example, the image of a bakery with the label "bakery" has a high similarity, while the image of a pet store with the label "bakery" has a low similarity, and so on.

[0042] Merchant profile: Information about the merchant obtained from the image, which can also be understood as the merchant's attribute information. For example, it may include the type of merchant (bread shop, pet store, etc.), the merchant's customer traffic density, the merchant's store size, or other attribute information of the merchant, etc.

[0043] It should be noted that the above-mentioned merchant profile acquisition scenario is only one of the multiple use cases provided in this specification. The image recognition method and system described in this specification can be applied not only to the merchant profile scenario but also to all image recognition scenarios, such as school profile information recognition, vehicle profile information recognition, or enterprise profile information recognition, etc. Those skilled in the art should understand that the image recognition method and system described in this specification applied to other use cases are also within the scope of protection of this specification.

[0044] Figure 1 This diagram illustrates an application scenario of an image recognition system 001 provided according to an embodiment of this specification. The image recognition system 001 (hereinafter referred to as system 001) can be applied to image recognition in any scenario, such as image recognition in school portrait information recognition scenarios, image recognition in vehicle portrait information recognition scenarios, image recognition in enterprise / organization portrait information recognition scenarios, etc. Figure 1 As shown, system 001 may include target user 100, client 200, server 300 and network 400.

[0045] The target user 100 can be the user who triggers the recognition of the target image, and the target user 100 can perform the image recognition operation on the client 200.

[0046] Client 200 can be a device for recognizing a target image in response to an image recognition operation by target user 100. In some embodiments, the image recognition method can be executed on client 200. In this case, client 200 may store data or instructions for performing the image recognition method described herein, and may execute or be used to execute said data or instructions. In some embodiments, client 200 may include a hardware device with data processing capabilities and the necessary programs required to drive the hardware device. Figure 1As shown, client 200 can communicate with server 300. In some embodiments, server 300 can communicate with multiple clients 200. In some embodiments, client 200 can interact with server 300 through network 400 to receive or send messages, such as receiving or sending training samples, such as image samples and text samples. In some embodiments, client 200 may include mobile devices, tablets, laptops, built-in devices in motor vehicles, or similar content, or any combination thereof. In some embodiments, the mobile device may include smart home devices, smart mobile devices, virtual reality devices, augmented reality devices, or similar devices, or any combination thereof. In some embodiments, the smart home device may include smart TVs, desktop computers, or any combination thereof. In some embodiments, the smart mobile device may include smartphones, personal digital assistants, gaming devices, navigation devices, or any combination thereof. In some embodiments, the virtual reality device or augmented reality device may include virtual reality headsets, virtual reality glasses, virtual reality patches, augmented reality headsets, augmented reality glasses, augmented reality patches, or similar content, or any combination thereof. For example, the virtual reality device or the augmented reality device may include Google Glass, head-mounted displays, VR, etc. In some embodiments, the built-in device in the motor vehicle may include an onboard computer, an onboard television, etc. In some embodiments, the client 200 may include an image acquisition device for acquiring video or image information of an object, thereby obtaining image samples or object images. In some embodiments, the image acquisition device may be a two-dimensional image acquisition device (such as an RGB camera), or it may be a combination of a two-dimensional image acquisition device (such as an RGB camera) and a depth image acquisition device (such as a 3D structured light camera, a laser detector, etc.). In some embodiments, the client 200 may be a device with positioning technology for locating the position of the client 200.

[0047] In some embodiments, the client 200 may have one or more applications (APPs) installed. The APP provides the target user 100 with the ability and interface to interact with the outside world via the network 400. The APPs include, but are not limited to: web browser APPs, search APPs, chat APPs, shopping APPs, video APPs, financial management APPs, instant messaging tools, email clients, social media platform software, etc. In some embodiments, the client 200 may have a target APP installed. The target APP can collect image samples or images of objects for the client 200. In some embodiments, the target object 100 can also trigger an image recognition request through the target APP. The target APP can respond to the image recognition request and execute the image recognition method described in this specification. The image recognition method will be described in detail later.

[0048] Server 300 may be a server providing various services, such as a backend server supporting target images acquired on client 200. In some embodiments, the image recognition method may be executed on server 300. In this case, server 300 may store data or instructions for executing the image recognition method described herein, and may execute or be used to execute said data or instructions. In some embodiments, server 300 may include hardware devices with data processing capabilities and necessary programs to drive the hardware devices. Server 300 may communicate with multiple clients 200 and receive data sent by clients 200.

[0049] Network 400 serves as a medium to provide a communication connection between client 200 and server 300. Network 400 facilitates the exchange of information or data. For example... Figure 1 As shown, client 200 and server 300 can connect to network 400 and transmit information or data to each other through network 400. In some embodiments, network 400 can be any type of wired or wireless network, or a combination thereof. For example, network 400 may include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigBee networks, near field communication (NFC) networks, or similar networks. In some embodiments, network 400 may include one or more network access points. For example, network 400 may include wired or wireless network access points, such as base stations or Internet switching points, through which one or more components of client 200 and server 300 can connect to network 400 to exchange data or information.

[0050] It should be understood that Figure 1 The number of clients 200, servers 300, and networks 400 shown is merely illustrative. Depending on implementation needs, there can be any number of clients 200, servers 300, and networks 400.

[0051] It should be noted that the image recognition method can be executed entirely on the client 200, entirely on the server 300, or partially on the client 200 and partially on the server 300.

[0052] Figure 2 A hardware structure diagram of a computing device 600 provided according to an embodiment of this specification is shown. The computing device 600 can perform the image recognition method described in this specification. The image recognition method is described in other parts of this specification. When the image recognition method is executed on a client 200, the computing device 600 can be the client 200. When the image recognition method is executed on a server 300, the computing device 600 can be the server 300. When the image recognition method can be executed partly on the client 200 and partly on the server 300, the computing device 600 can be both the client 200 and the server 300.

[0053] like Figure 2 As shown, the computing device 600 may include at least one storage medium 630 and at least one processor 620. In some embodiments, the computing device 600 may also include a communication port 650 and an internal communication bus 610. Additionally, the computing device 600 may include I / O components 660.

[0054] The internal communication bus 610 can connect different system components, including storage medium 630, processor 620 and communication port 650.

[0055] I / O component 660 supports input / output between computing device 600 and other components.

[0056] Communication port 650 is used for data communication between computing device 600 and external sources. For example, communication port 650 can be used for data communication between computing device 600 and network 400. Communication port 650 can be a wired communication port or a wireless communication port.

[0057] Storage medium 630 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 632, a read-only storage medium (ROM) 634, or a random access storage medium (RAM) 636. Storage medium 630 also includes at least one instruction set stored in the data storage device. The instructions are computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that perform the image recognition methods provided in this specification.

[0058] At least one processor 620 can be communicatively connected to at least one storage medium 630 and a communication port 650 via an internal communication bus 610. The at least one processor 620 is used to execute the at least one instruction set described above. When the computing device 600 is running, the at least one processor 620 reads the at least one instruction set and, according to the instructions of the at least one instruction set, executes the image recognition method provided in this specification. The processor 620 can execute all the steps included in the image recognition method. The processor 620 can be in the form of one or more processors. In some embodiments, the processor 620 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or any combination thereof. For illustrative purposes only, only one processor 620 is described in this specification for the computing device 600. However, it should be noted that the computing device 600 in this specification may also include multiple processors. Therefore, the operation and / or method steps disclosed in this specification may be executed by one processor as described in this specification, or they may be executed jointly by multiple processors. For example, if the processor 620 of the computing device 600 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 620 (e.g., the first processor executes step A, the second processor executes step B, or the first and second processors jointly execute steps A and B).

[0059] Figure 3A flowchart of an image recognition method P100 according to an embodiment of this specification is shown. As previously described, computing device 600 can execute the image recognition method P100 of this specification. Specifically, processor 620 can read an instruction set stored in its local storage medium and then execute the image recognition method P100 of this specification according to the instructions in the instruction set. Figure 3 As shown, method P100 may include:

[0060] S110: Obtain the training sample set and extract at least one training sample pair from the training sample set.

[0061] The training sample set includes a full set of samples from all objects and a test set of samples from the target objects, with the full set containing more samples than the test set. The full set can be understood as a publicly available dataset containing text samples with various attributes (portrait information) and image samples of various objects corresponding to each attribute. The test samples can be understood as containing only a small number of known attributes of the target objects and image samples related to them. The full set of objects may or may not include the target objects. The full set ensures the cold-start capability of the recognition model, allowing it to recognize not only known attributes of the target objects but also newly added attributes without requiring retraining. The test set ensures the recognition model's performance in recognizing attribute information of the target objects. Therefore, specific algorithms can be used to sample from these two sets, achieving a trade-off between cold-start capability and performance.

[0062] There are several ways to obtain the training sample set, including the following:

[0063] For example, the processor 620 can receive a training sample set uploaded by the target user 100 through a terminal or other device, or it can obtain text samples of attribute information of at least one candidate object and at least one image sample corresponding to the attribute information from a network or multimodal sample database to obtain a full sample set. From the full sample set, a preset number of text samples and image samples corresponding to the target object are selected to obtain a test sample set. Alternatively, the processor 620 can also select text samples and image samples corresponding to each attribute information from the network or multimodal sample database by sampling with or without replacement to obtain a full sample set and a test sample set.

[0064] After obtaining the training sample set, at least one training sample pair can be extracted from the training sample set. Each training sample pair includes an image sample and a text sample. There are several ways to extract at least one training sample pair. For example, the processor 620 can extract at least one candidate sample pair from the full sample set, extract at least one test sample pair from the test sample set, or use at least one candidate sample pair and at least one test sample pair as at least one training sample pair.

[0065] Each candidate sample pair may include text samples and image samples corresponding to the attribute information of the candidate object. Each test sample pair may include text samples and image samples corresponding to the attribute information of the target object. Furthermore, each attribute information may correspond to one or more text samples, or one or more image samples; therefore, the image or text samples in different training sample pairs may be the same. Finally, the number of candidate sample pairs and test samples extracted can be arbitrary, and may be the same or different.

[0066] S120: Use a recognition model to extract features from each training sample pair to obtain the image features of the image sample and the text features of the text sample.

[0067] Image features can be characteristic information representing image samples, and text features can be characteristic information representing text samples.

[0068] The recognition model can include an encoding network and a sampling network. The encoding network is mainly used to encode features from image samples and text samples to obtain image features and text features. The encoding network can be a multimodal encoder, which can include an image encoder and a text encoder. The image encoder is used to encode features from image samples to output image features, and the text encoder is used to encode features from text samples to output text features.

[0069] There are several ways to use a recognition model to extract features from each training sample pair, as follows:

[0070] For example, the processor 620 can extract image samples and text samples from each training sample pair, encode the image samples using the image encoder of the encoding network to obtain image features, and encode the text samples using the text encoder of the encoding network to obtain text features.

[0071] The network structure of the encoding network can be varied. For example, it can be a multimodal encoder similar to CLIP (a multimodal network structure that connects text and images), or it can be other network structures that can encode features of multimodal information such as images / text.

[0072] S130: Extract at least one image sub-feature corresponding to a feature region of a preset size from the image features, and identify the text sub-feature corresponding to each text word in the text sample from the text features.

[0073] The feature region can be a region in the feature map of the image features, or it can be understood as the region in the feature map corresponding to a certain region in the image sample.

[0074] In this context, text words can be understood as words or phrases obtained by segmenting a text sample. For example, if the text sample is "large pet store", then the text sample can include words or phrases such as "large" and "pet store", which can be considered text words.

[0075] There are several ways to extract image sub-features from image features and identify text sub-features from text features, as follows:

[0076] For example, the processor 620 can obtain the similarity between image features and text features, obtain a global similarity set corresponding to at least one training sample pair, select a target training sample pair from the at least one training sample pair based on the global similarity set, and extract at least one image sub-feature corresponding to a feature region of a preset size from the image features based on the target training sample pair, and identify the text sub-feature corresponding to each text word in the text sample from the text features. Specifically, it can be as follows:

[0077] S131: Obtain the similarity between image features and text features to obtain at least one global similarity set corresponding to training sample pairs.

[0078] The global similarity set can include at least one global similarity, which can be understood as the overall similarity between image features and text features.

[0079] There are several ways to obtain the similarity between image features and text features, including the following:

[0080] For example, the processor 620 can calculate the cosine similarity between image features and text features, and use the cosine similarity as the global similarity between image features and text features, thereby obtaining a global similarity set corresponding to at least one training sample pair. Alternatively, it can calculate the feature distance between image features and text features, and use the feature distance as the global similarity between image features and text features, thereby obtaining a global similarity set for at least one training sample.

[0081] There are various types of feature distances, such as Euclidean distance or other types of feature distances.

[0082] S132: Based on the global similarity set, select the target training sample pair from at least one training sample pair.

[0083] In this context, the target training sample pair can be understood as the training sample pair used to train or converge the encoding network of the recognition model.

[0084] There are several ways to select the target training sample pair from at least one training sample pair based on the global similarity set, as follows:

[0085] For example, the processor 620 can determine the global loss information corresponding to each training sample pair based on global similarity, determine the sampling probability of each training sample pair based on the image features, text features and global loss information corresponding to each training sample pair, and select the target training sample pair from at least one training sample pair based on the sampling probability.

[0086] The global loss information can be understood as the information lost by the encoder. There are multiple ways to determine the global loss information corresponding to each training sample based on the global similarity. For example, the processor 620 can filter out the similarity corresponding to each training sample pair in the global similarity set to obtain the target global similarity. Based on the target global similarity, it can obtain the feature difference between the image features and text features corresponding to each training sample pair, and based on the feature difference, determine the global loss information corresponding to each training sample pair.

[0087] There are several ways to determine the global loss information corresponding to each training sample pair based on the feature difference. For example, the processor 620 can use the L2 loss function to calculate the sum of squares of the feature differences, thereby obtaining the global loss information corresponding to each training sample pair, as shown in formula (1):

[0088]

[0089] Among them, Loss clip For global loss information, f img For image features, f text For text features.

[0090] After determining the global loss information for each training sample pair, the sampling probability of that training sample pair can be determined based on the global loss information, image features, and text features. The sampling probability can be understood as the probability information of using that training sample pair as the target training sample pair for training or convergence of the encoding network. There are several ways to determine the sampling probability of each training sample pair. For example, the processor 620 can determine the gradient corresponding to each training sample pair based on the global loss information, and determine the sampling probability of each training sample pair based on the image features, text features, and gradient corresponding to the recognition model's sampling network.

[0091] The gradient can be understood as the direction of descent or the information of the extreme point of the function when the encoding network converges based on the global loss information. It can usually be obtained by directly differentiating the global loss information. There are multiple ways to determine the gradient corresponding to each training sample based on the global loss information. For example, the processor 620 can differentiate the global loss information to obtain the gradient corresponding to the training sample, as shown in formula (2).

[0092]

[0093] Where x is a training sample pair, grad x The gradient corresponding to this training sample pair. This is global loss information.

[0094] After determining the gradient corresponding to each training sample pair, the sampling network of the recognition model can determine the sampling probability of each training sample pair based on the image features, text features, and gradient corresponding to each training sample pair. There are several ways to determine the sampling probability. For example, the processor 620 can input the image features, text features, and gradient corresponding to each training sample pair into the sampling network, and the sampling network can output the sampling probability of that training sample pair.

[0095] The sampling network can have various network structures, such as meta-networks or other network structures that can output sampling probabilities through gradients, image features, and text features, etc.

[0096] After determining the sampling probability of each training sample pair, there are several ways to select target training samples from at least one training sample pair based on the sampling probability. For example, the processor 620 can select training sample pairs from at least one training sample pair whose sampling probability exceeds a preset probability threshold to obtain target training sample pairs. Alternatively, it can sort each training sample pair in at least one training sample pair based on the sampling probability, and select the TOP K training sample pairs from at least one training sample pair as target training sample pairs based on the sorting result. Or, it can classify at least one training sample pair according to the object type corresponding to the training sample pair to obtain a set of training sample pairs corresponding to each object type, sort the training sample pairs in the set of training sample pairs based on the sampling probability, and select target training sample pairs from the set of training sample pairs based on the sorting result. Alternatively, it can obtain the weighting coefficient corresponding to each object type based on the object type corresponding to the training sample pair, weight the sampling probability of the training sample pair based on the weighting coefficient, and select target training sample pairs from at least one training sample pair based on the weighted sampling probability.

[0097] In the training process of the recognition model, sampling more candidate sample pairs from the full sample set can increase the cold-start capability of the recognition model, while sampling more test sample pairs from the test sample set can improve the recognition performance of the recognition model for the portrait information of the target object. In this scheme, the sampling probability of each training sample pair can be determined by a sampling network, and the target training sample is selected from at least one training sample pair based on the sampling probability. Thus, the sampling strategy of reinforcement learning can balance the sampling quantity between the full sample set and the test sample set, so that the recognition model can achieve a trade-off between cold-start capability and performance.

[0098] S133: Based on the target training sample pair, extract at least one image sub-feature corresponding to the feature region of a preset size from the image features, and identify the text sub-feature corresponding to each text word in the text sample from the text features.

[0099] Among them, the target-based training sample pairs include target image samples and target text samples.

[0100] There are several ways to extract image sub-features from image features and identify text sub-features from text features, as follows:

[0101] For example, the processor 620 can filter out target image features of target image samples from image features, filter out target text features of target text samples from text features, extract at least one image sub-feature corresponding to a feature region of a preset size from target image features, and extract text sub-features corresponding to each text word in target text samples from target text features.

[0102] There are several ways to extract at least one image sub-feature corresponding to a feature region of a preset size from the target image features. For example, the processor 620 can traverse the feature region of a preset size in the feature map corresponding to the target image features and extract the image features corresponding to each feature region in the target image features, thereby obtaining at least one image sub-feature.

[0103] The preset size can be any value, which can be set according to the size of the feature map or the actual application. For example, it can be 5*5 or other sizes.

[0104] There are several ways to extract text sub-features of each text word pair in the target text sample from the target text features. For example, the processor 620 can perform text segmentation on the target text sample to obtain at least one text word in the target text sample, and then filter out the text features corresponding to each text word in the at least one text word from the target text features to obtain the text sub-features corresponding to each text word.

[0105] There are multiple ways to perform text segmentation on the target text sample. For example, the processor 620 can obtain the text type of the text sample, use the segmentation algorithm corresponding to the text type to perform text segmentation on the target text sample, and thus obtain at least one text word in the target text sample.

[0106] The text type can include at least one of long text, short text, Chinese text, and non-Chinese text.

[0107] S140: Traverse the similarity between at least one image sub-feature and text sub-feature to obtain a local similarity set, and converge the recognition model based on the local similarity set to obtain the target recognition model.

[0108] The local similarity set includes at least one local similarity. Local similarity can be understood as the similarity between a region in an image sample and a word in a text sample, or as the similarity between a sub-feature of an image feature and a sub-feature of a word in a text feature. Compared to global similarity, local similarity can constrain multimodal relationships with more fine-grained precision.

[0109] S141: Traverse the similarity between at least one image sub-feature and the text sub-feature to obtain a local similarity set.

[0110] There are several ways to iterate through the similarity between at least one image sub-feature and text sub-feature, as follows:

[0111] For example, the processor 620 can traverse each image sub-feature and each text sub-feature in at least one image sub-feature, calculate the similarity between each image sub-feature and each text sub-feature respectively, and use the similarity as a local similarity to obtain a set of local similarities.

[0112] S142: The recognition model is converged based on the local similarity set to obtain the target recognition model.

[0113] For example, the processor 620 can perform a preset number of encoding training operations on the encoding network based on the sampling network, global loss information, and local similarity set to obtain a target encoding network. Based on the target encoding network, the sampling network can be trained on a preset number of sampling operations to obtain a target sampling network. The target encoding network and the target sampling network are used as the current recognition model, and the recognition model is updated to the current recognition model. Then, the processor returns to the step of using the recognition model to extract features from each training sample pair until the recognition model converges to obtain the target recognition model. Specifically, this can be done as follows:

[0114] (1) Based on the sampling network, global loss information and local similarity set, the encoding network is trained for a preset number of times to obtain the target encoding network.

[0115] In this context, encoding training can be understood as updating the network parameters in the encoding network. Based on the sampled network, global loss information, and local similarity set, there are several ways to perform a preset number of encoding training operations on the encoding network, as detailed below:

[0116] For example, the processor 620 can determine the encoding loss information of the encoding network based on the global loss information and the local similarity set, update the network parameters of the encoding network based on the encoding loss information to obtain the current encoding network, use the current encoding network and the sampling network as candidate recognition models, update the recognition model to the candidate recognition model, and return to execute the step of using the recognition model to extract features from each training sample pair until the number of updates reaches the preset first number, thus obtaining the target encoding network.

[0117] The encoding loss information can be understood as the loss information of the encoding network when encoding features of image samples and text samples. There are multiple ways to determine the encoding loss information of the encoding network based on global loss information and local similarity sets. For example, the processor 620 can filter out the target global loss information corresponding to the target training sample pairs from the global loss information, determine the corresponding local loss information of the encoding network based on the local similarity set, and fuse the global loss information and local loss information to obtain the encoding loss information of the encoding network.

[0118] Local loss information can be understood as the loss information between local regions of an image sample and text words of a text sample. Based on the local similarity set, there are multiple ways to determine the local loss information corresponding to the encoding network. For example, the processor 620 can traverse the local similarity set to find the local similarity corresponding to each text sub-feature, obtain a candidate local similarity set for each text sub-feature, select the local similarity with the highest similarity from the candidate local similarity set to obtain the target local similarity corresponding to each text sub-feature, and adjust the target local similarity to a preset similarity to obtain the local loss information corresponding to the encoding network.

[0119] There are several ways to adjust the target local similarity to the preset similarity, where the preset similarity is greater than the target local similarity. For example, the processor 620 can directly adjust the target local similarity to the preset similarity and determine the local loss information corresponding to the encoding network based on the adjusted local similarity. Alternatively, it can adjust the image sub-features and text sub-features corresponding to the target local similarity based on the preset similarity, so that the local similarity between the adjusted image sub-features and text sub-features reaches the preset similarity. The local loss information corresponding to the encoding network is then determined based on the adjusted image sub-features and adjusted text sub-features.

[0120] The preset similarity can be the maximum similarity set in actual application, for example, it can be 1 or any other value. Adjusting the target local similarity can be understood as maximizing the target local similarity. The so-called maximization can be understood as maximizing the local similarity between the largest region and text words, as shown in formula (3):

[0121]

[0122] Among them, Loss local For local loss information, fm i For the i-th image sub-feature, text j Let be the j-th text sub-feature.

[0123] After determining the local loss information corresponding to the coding network, the global loss information and the local loss information can be fused to obtain the coding loss information of the coding network. There are several ways to fuse the global loss information and the local loss information. For example, the processor 620 can directly concatenate or combine the global loss information and the local loss information to obtain the coding loss information of the coding network. Alternatively, it can obtain the weighting coefficients of the global loss information and the local loss information separately, and then weight the global loss information and the local loss information separately based on the weighting coefficients. Finally, it can fuse the weighted global loss information and the weighted local loss information to obtain the coding loss information of the coding network.

[0124] After determining the coding loss information of the coding network, the network parameters of the coding network can be updated based on the coding loss information to obtain the current coding network. There are several ways to update the network parameters of the coding network based on the coding loss information. For example, the processor 620 can use the gradient descent algorithm to update the network parameters of the coding network based on the coding loss information to obtain the current coding network. Alternatively, the processor 620 can also use other network parameter update algorithms to update the network parameters of the coding network based on the coding loss information to obtain the current coding network.

[0125] After updating the network parameters of the encoding network, the current encoding network and the sampling network can be used as candidate recognition models, and the recognition model is updated to the candidate recognition model. Then, the processor 620 can return to the step of extracting features from each training sample pair using the recognition model, until the number of updates reaches the preset first number, and the final encoding network and sampling network can be used as the target encoding network.

[0126] (2) Based on the target encoding network, the sampling network is trained by a preset number of samplings to obtain the target sampling network.

[0127] Sampling training can be understood as updating the network parameters of the sampling network. Based on the target encoding network, the sampling network is trained with a preset number of sampling iterations. There are several ways to obtain the target sampling network, such as as follows:

[0128] For example, the processor 620 can use a target encoding network to extract features from at least one training sample pair to obtain a current global loss information set corresponding to at least one training sample pair. Based on the sampling network, candidate global loss information is selected from the current global loss information set, and target test loss information is extracted from the candidate global loss information. Based on the target test loss information, the network parameters of the sampling network are updated to obtain the current sampling network. The sampling network is then updated to the current sampling network, and the process returns to the step of using the target encoding network to extract features from at least one training sample pair until the number of updates reaches a preset second number, thus obtaining the target sampling network.

[0129] There are several ways to use a target encoding network to extract features from at least one training sample pair to obtain the current global loss information set corresponding to at least one training sample pair. For example, the processor 620 can use a target encoding network to extract features from each of the at least one training sample pair to obtain a sample feature pair corresponding to each training sample pair. The sample feature pair includes image features and text features. Then, the similarity between the image features and text features in the sample feature pair is calculated. Based on the similarity, the global loss information corresponding to the sample feature pair is determined, thereby obtaining the current global loss information set corresponding to at least one training sample pair.

[0130] After obtaining the current global loss information set corresponding to at least one training sample pair, candidate global loss information can be selected from the current global loss information set based on the sampling network. The method of selecting candidate global loss information from the current global loss information by sampling probability based on the sampling network is similar to the method of selecting the target global loss, as described above, and will not be repeated here.

[0131] After selecting candidate global loss information, the target test loss information can be extracted from the candidate global loss information. The target test loss information can be understood as the loss information corresponding to the test sample pair in the candidate global loss information, or as the loss information of the sampled data on the test set. There are multiple ways to extract the target test loss information from the candidate global loss information. For example, the processor 620 can select at least one global loss information corresponding to the test sample pair from the candidate global loss information to obtain a test loss information set, and select the smallest test loss information from the test loss information set to obtain the target test loss information, as shown in formula (4):

[0132] Loss meta =min(s(X)) (4)

[0133] Among them, Loss metaLet s(X) be the target test loss information, and s(X) be the global loss information corresponding to the test sample pair.

[0134] After extracting the target test loss information, the network parameters of the sampling network can be updated based on the target test loss information. The update method is similar to that of the encoding network, so it will not be described in detail here, thus obtaining the current sampling network.

[0135] After obtaining the current sampling network, the sampling network can be updated to the current sampling network, and the step of extracting features from at least one training sample using the target encoding network can be performed until the number of updates reaches the preset second number, thus obtaining the target sampling network.

[0136] After training the encoding network and the sampling network, the trained target encoding network and target sampling network are used as the current recognition model, and the recognition model is updated to the current recognition model. The processor 620 can then return to the step of extracting features from each training sample pair using the recognition model, until the recognition model converges, resulting in the target recognition model.

[0137] There are various convergence conditions for the recognition model. For example, the number of training iterations of the recognition model can reach a preset target number, the performance of the recognition model can reach a preset performance level, or the size of the recognition model can reach a preset size.

[0138] Training the recognition model once can be viewed as training the encoding network for a preset number of iterations and the sampling network for a preset number of iterations, but the training processes for the two are independent. The preset first iteration number is greater than the preset second iteration number, meaning that before the recognition model converges, the encoding network is trained more times than the sampling network. It should also be noted that the encoding network and sampling network are trained independently; that is, when training the encoding network, the sampling network is fixed, and vice versa. Furthermore, the ratio between the preset first iteration number and the preset second iteration number is a preset ratio, which can be set according to the actual application, for example, 10:1, or any other arbitrary ratio. Taking a preset ratio of 10:1, with the encoding network as the encoder and the sampling network as the meta-network as an example, after each training iteration of the recognition model, 10 rounds of encoder training can be performed first, followed by one round of meta-network training, until the recognition model converges, thus obtaining the trained target recognition model.

[0139] In some embodiments, before training the recognition model, the preset recognition model can be pre-trained using image-text relationships to obtain the recognition model. Therefore, before using the recognition model to extract features from each training sample pair, the processor 620 can pre-train the preset recognition model to obtain the recognition model. There are various pre-training methods. For example, the processor 620 can use the sampling network in the preset recognition model and at least one training sample pair to perform a preset first number of encoding training operations on the encoding network in the preset recognition model to obtain a basic encoding network. Based on the basic encoding network and at least one training sample pair, it can perform a preset second number of sampling training operations on the sampling network in the preset recognition model to obtain a basic sampling network. The basic encoding network and the basic sampling network are used as the basic recognition model, and the preset recognition model is updated to the basic recognition model. The processor then returns to the step of performing a preset first number of encoding training operations on the encoding network in the preset recognition model using the sampling network in the preset recognition model and at least one training sample pair, until the preset recognition model converges, thus obtaining the recognition model.

[0140] The preset recognition model can be an initial recognition model that has not undergone image-text relationship pre-training. The preset recognition model may include an encoding network and a sampling network. There are various ways to perform a preset first-round encoding training on the encoding network of the preset recognition model using the sampling network and at least one training sample pair. For example, the processor 620 can use the encoding network of the preset recognition model to extract features from at least one training sample pair to obtain a candidate global loss information set corresponding to at least one training sample pair. Based on the sampling network of the preset recognition model, it can select basic global loss information from the candidate global loss information set, and then perform a preset first-round encoding training on the encoding network of the preset recognition model based on the basic global loss information to obtain the basic encoding network.

[0141] In this process, the encoding network in the preset recognition model is used to extract features from at least one training sample pair to obtain candidate global loss information corresponding to at least one training sample pair. This is done in the same way as obtaining the set of global loss information, as detailed above. It will not be repeated here.

[0142] The method by which the sampling network based on the preset recognition model selects basic global loss information from the candidate global loss information set is the same as the method by which it selects target global loss information from the global loss information set, as detailed above, and will not be repeated here. Furthermore, the main difference between the encoding training of the encoding network in the preset recognition model and the encoding training of the encoding network in the recognition model is that the loss information in the encoding training of the preset recognition model is only basic global loss information and does not include local loss information. Therefore, the encoding training of the encoding network in the preset recognition model focuses on the pre-training of image-text relationships, while the encoding training of the encoding network in the recognition model focuses on training the fine-grained relationships between image regions and text words. This allows for a gradual improvement in the recognition granularity of the recognition model and increases its recognition accuracy.

[0143] After training the encoding network in the preset recognition model, the sampling network in the preset recognition model can be trained a second time using the trained base encoding network and at least one training sample pair, thus obtaining the trained base sampling network. The training of the sampling network in the preset recognition model is similar to the training of the sampling network in the recognition model, as detailed above, and will not be repeated here.

[0144] The preset first and second numbers in a complete training run of the preset recognition model can be set according to the actual application; they can be the same or different. The ratio between the preset first and second numbers in the preset recognition model training can also be preset, with the preset first number being greater than the preset second number.

[0145] After training the encoding network and sampling network in the preset recognition model to obtain the basic recognition model, the recognition model can be updated to the basic recognition model. Then, the processor 620 can return to the step of performing a preset first number of encoding training on the encoding network in the preset recognition model using the sampling network in the preset recognition model and at least one training sample pair, until the preset recognition model converges, thus obtaining the recognition model.

[0146] The training of the preset recognition model and the training of the recognition model are similar in the overall training process. The main difference lies in the encoding training of the encoding network in the training of the preset recognition model. The loss information used in the encoding training of the preset recognition model is global loss information, while the loss information in the encoding training of the recognition model can include both global and local loss information. This allows for a gradual increase in the recognition granularity and accuracy of the recognition model.

[0147] In some embodiments, after the recognition model converges based on the local similarity set to obtain the target recognition model, the attribute information (object profile) of the target object can be defined, thereby enabling the target recognition model to accurately identify the attribute information of the target object in the object image. Therefore, the processor 620 can also obtain a text set corresponding to at least one preset attribute information of the target object, and use the target recognition model to extract features from each text in the text set to obtain a text feature set corresponding to at least one preset attribute information. This text feature set can then be used to match the image features of the object image, and the preset attribute information corresponding to the successfully matched text features can be used as the attribute information of the target object corresponding to the object image.

[0148] Among them, the preset attribute information can be the commonly used attribute information of the target object that is set in advance according to business needs. Taking the target object as a merchant as an example, the preset attribute information can include merchant category, store size, store traffic information or store address, etc.

[0149] S150: The target recognition model is used to identify the attribute information of the target object in the object image.

[0150] In this context, an object image can be understood as an image containing a target object. A target object can be understood as one or more specific objects pre-defined according to business needs. For example, in a payment business, the target object could be a merchant, and in a school setting, the target object could be a student or a teacher, and so on.

[0151] Among them, attribute information can be understood as relevant information that represents the attributes of the target object. For example, taking the target object as a merchant, the attribute information may include merchant category, store size, store traffic information or store address, etc.

[0152] There are several ways to use a target recognition model to identify the attribute information of a target object in an object image, as follows:

[0153] For example, the processor 620 can acquire an object image of the target object, use a target recognition model to extract features from the object image to obtain the current image features of the object image, and determine the attribute information of the target object in the object image based on the text feature set and the current image features.

[0154] There are several ways to obtain the object image of the target object. For example, the processor 620 can directly obtain the object image of the target object uploaded by the user through the terminal, or it can obtain the object image of at least one target object from the image database, or it can receive an image recognition request, which includes the storage address of the object image of the target object, and obtain the object image of the target object based on the storage address.

[0155] After obtaining the object image of the target object, a target recognition model can be used to extract features from the object image to obtain the current image features. There are multiple ways to extract features from the object image. For example, the processor 620 can use the encoding network in the target recognition model to extract image features from the object image to obtain the current image features of the object image.

[0156] After extracting the current image features of the object image, the attribute information of the target object in the object image can be determined based on the text feature set and the current image features. There are multiple ways to determine the attribute information of the target object in the target image. For example, the processor 620 can obtain the similarity between the current image features and each text feature in the text feature set to obtain the current similarity set, filter out the target similarity exceeding a preset similarity threshold in the current similarity set, and filter out the attribute information corresponding to the target similarity from at least one attribute information to obtain the attribute information of the target object.

[0157] There are several ways to obtain the similarity between the current image features and each text feature in the text feature set. For example, the processor 620 can directly calculate the cosine similarity or feature distance between the current image features and each text feature in the text feature set, and use the cosine similarity or feature distance as the similarity between the current image features and each text feature in the text feature set to obtain the current similarity set. Alternatively, it can traverse the current image features to find at least one current image sub-feature corresponding to a feature region of a preset size, and identify the current text sub-feature corresponding to the text word in each text feature in the text feature set. Then, it can calculate the similarity between the current image sub-feature and the current text sub-feature respectively, select the largest similarity from the calculated similarities, and obtain the target similarity between the text sub-features matched by the current image sub-feature. The target similarities are then fused to obtain the current similarity between the current image features and each text feature. Finally, the current similarity set is obtained by fusing the current similarities.

[0158] The preset similarity threshold can be a pre-defined threshold for judging whether an image sample matches preset attribute information. This preset similarity threshold can be set according to the actual application. Taking a preset similarity threshold of T and a merchant as the target object as an example, the preset attribute information corresponding to text with a similarity exceeding T can be the merchant's attributes (merchant profile).

[0159] In some embodiments, when new attribute information is added in addition to preset attribute information, direct adaptation can be performed. Direct adaptation relies on the cold-start capability of the recognition model. Therefore, there are multiple ways to perform cold-start for new attribute information. For example, the processor 620 can obtain the target text corresponding to the current attribute information, update at least one attribute information based on the current attribute information to obtain an updated attribute information set, use a target recognition model to extract features from the target text to obtain the current text features, and update the text feature set based on the current text features to obtain an updated text feature set corresponding to the attribute information set. Based on the updated text feature set, the attribute information of the target object in the object image is identified.

[0160] There are several ways to identify the attribute information of a target object in an object image based on the updated text feature set. For example, the processor 620 can use the encoding network of a target recognition model to extract features from the object image, obtain image features, and extract the text features corresponding to the newly added target attributes from the updated text feature set. When the similarity between the image features and the text features exceeds a preset similarity threshold, the newly added attribute information can be added to the attribute information of the target object, indicating that the object image can include the newly added attribute information of the target object. Furthermore, the preset similarity threshold for the newly added attribute information can be the same as or different from the preset similarity threshold for the corresponding preset attribute information.

[0161] Among them, the cold start method for the newly added attribute information does not require retraining of the target recognition model, which greatly reduces the adaptation and upgrade costs for the new attribute information and can also improve the recognition efficiency of image recognition.

[0162] Taking merchants as an example, image recognition can be understood as extracting a merchant profile from a merchant image. This scheme differs from traditional image classification methods in that it can obtain merchant profiles based on fine-grained multi-model relationships. The specific acquisition process can be described as follows: Figure 4As shown, the training of the recognition model can be divided into two training stages, mainly including the first-stage image-text relationship pre-training and the two-node fine-grained relationship training. After training the recognition model, the portrait attributes can be defined and obtained. When adding new attribute information, a cold start can be performed on the new attributes. In the first-stage image-text relationship pre-training, the sampling technique of meta-networks can be used to perform balanced sampling in the public dataset and the merchant dataset, and the image-text relationship model is trained based on the sampled training samples. In the second-stage fine-grained relationship training, constraints on local regions (feature regions) and fields (text words) are added to achieve fine-grained relationship training. When obtaining portrait attribute information, the required portrait attributes are defined, and then the trained target recognition model is used to directly obtain the attribute information. When adding new attributes, a cold start method can be directly used for image recognition.

[0163] In summary, the image recognition method P100 and system 001 provided in this specification acquire a training sample set, and extract at least one training sample pair from the training sample set. Each training sample pair includes an image sample and a text sample. A recognition model is used to extract features from each training sample pair to obtain image features of the image sample and text features of the text sample. Then, at least one image sub-feature corresponding to a feature region of a preset size is extracted from the image features, and the text sub-feature corresponding to each text word in the text sample is identified from the text features. The process iterates between at least one image sub-feature and the text sub-feature. The similarity is used to obtain a local similarity set, and the recognition model is converged based on the local similarity set to obtain the target recognition model. The target recognition model is then used to identify the attribute information of the target object in the object image. Since this scheme extracts image sub-features from image features and extracts text sub-features of each text word in the text sample during the training process of the recognition model, and then iterates through the similarity between image sub-features and text sub-features to constrain them, the granularity of the constraint is refined to the relationship between feature regions and text words, thereby improving the recognition accuracy of the recognition model. Therefore, the recognition accuracy of image recognition can be improved.

[0164] This specification, in another aspect, provides a non-transitory storage medium storing at least one set of executable instructions for performing image recognition. When the executable instructions are executed by a processor, they instruct the processor to implement the steps of the image recognition method P100 described herein. In some possible embodiments, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on a computing device 600, the program code causes the computing device 600 to perform the steps of the image recognition method P100 described herein. The program product for implementing the above method may employ a portable compact disc read-only memory (CD-ROM) containing program code and may run on the computing device 600. However, the program product of this specification is not limited thereto. In this specification, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on computing device 600, partially on computing device 600, as a standalone software package, partially on computing device 600 and partially on a remote computing device, or entirely on a remote computing device.

[0165] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0166] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0167] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0168] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0169] Each patent, patent application, publication of the patent application, and other materials such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference. All contents used for all purposes, except for any history of prosecution documents relating to it, that may be inconsistent with or conflict with this document, or any such history of prosecution documents that may have a limiting effect on the widest extent of the claims, are now or hereafter associated with this document. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any of the included materials and the terms, description, definition, and / or used in connection with this document, the terms used herein shall prevail.

[0170] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. An image recognition method, comprising: Obtain a training sample set, wherein the training sample set includes a full sample set and a test sample set of the target object, the full sample set includes text samples corresponding to each attribute information and object image samples of various types of objects, and the number of samples in the full sample set is greater than the number of samples in the test sample set; At least one candidate sample pair is extracted from the full sample set; at least one test sample pair is extracted from the test sample set; the at least one candidate sample pair and the at least one test sample pair are used as at least one training sample pair, each of the at least one training sample pair includes an image sample and a text sample; The recognition model is used to extract features from each training sample pair to obtain the image features of the image sample and the text features of the text sample; Extract at least one image sub-feature corresponding to a feature region of a preset size from the image features, and identify the text sub-feature corresponding to each text word in the text sample from the text features; By iterating through the similarities between at least one image sub-feature and the text sub-feature, a local similarity set is obtained. The recognition model is then converged based on this local similarity set to obtain a target recognition model. The target recognition model is used to identify the attribute information of the target object in the object image.

2. The image recognition method according to claim 1, wherein, The step of extracting at least one image sub-feature corresponding to a feature region of a preset size from the image features, and identifying the text sub-feature corresponding to each text word in the text sample from the text features, includes: Obtain the similarity between the image features and the text features to obtain the global similarity set corresponding to the at least one training sample pair; Based on the global similarity set, target training sample pairs are selected from the at least one training sample pair; and Based on the target training sample pair, at least one image sub-feature corresponding to a feature region of a preset size is extracted from the image features, and the text sub-feature corresponding to each text word in the text sample is identified in the text features.

3. The image recognition method according to claim 2, wherein, The target training sample pair includes target image samples and target text samples; and The step of extracting at least one image sub-feature corresponding to a feature region of a preset size from the image features based on the target training sample pair, and identifying the text sub-feature corresponding to each text word in the text sample from the text features, includes: The target image features of the target image sample are selected from the image features, and the target text features of the target text sample are selected from the text features; Extract at least one image sub-feature corresponding to a feature region of a preset size from the target image features; and Extract the text sub-features corresponding to each text word in the target text sample from the target text features.

4. The image recognition method according to claim 2, wherein, The step of selecting target training sample pairs from the at least one training sample pair based on the global similarity set includes: Based on the global similarity set, determine the global loss information corresponding to each training sample pair; Based on the image features, text features, and global loss information corresponding to each training sample pair, a sampling probability for each training sample pair is determined, wherein the sampling probability represents the probability information of each training sample as a target training sample pair; and Based on the sampling probability, a target training sample pair is selected from the at least one training sample pair.

5. The image recognition method according to claim 4, wherein, The step of determining the global loss information corresponding to each training sample pair based on the global similarity set includes: The similarity corresponding to each training sample pair is selected from the global similarity set to obtain the target global similarity. Based on the target global similarity, the feature difference between the image features and the text features corresponding to each training sample pair is obtained; and Based on the feature differences, the global loss information corresponding to each training sample pair is determined.

6. The image recognition method according to claim 4, wherein, The step of determining the sampling probability of each training sample pair based on the image features, text features, and global loss information corresponding to each training sample pair includes: Based on the global loss information, determine the gradient corresponding to each training sample pair; and The sampling probability of each training sample pair is determined by the sampling network of the recognition model based on the image features, text features, and gradient corresponding to each training sample pair.

7. The image recognition method according to claim 4, wherein, The recognition model includes an encoding network and a sampling network; as well as The process of converging the recognition model based on the local similarity set to obtain the target recognition model includes: Based on the sampling network, the global loss information, and the local similarity set, the encoding network is trained for a preset number of first encoding iterations to obtain the target encoding network. Based on the target encoding network, the sampling network is trained with a preset number of samplings a second time to obtain the target sampling network; The target encoding network and the target sampling network are used as the current recognition model, and the recognition model is updated to the current recognition model; and Return to the step of using the recognition model to extract features from each training sample pair until the recognition model converges, thus obtaining the target recognition model.

8. The image recognition method according to claim 7, wherein, The preset first number is greater than the second preset number, and the ratio between the preset first number and the preset second number is a preset ratio.

9. The image recognition method according to claim 7, wherein, The step of performing a preset number of encoding training iterations on the encoding network based on the sampling network, the global loss information, and the local similarity set to obtain the target encoding network includes: Based on the global loss information and the local similarity set, the coding loss information of the coding network is determined; The network parameters of the coding network are updated based on the coding loss information to obtain the current coding network; The current encoding network and sampling network are used as candidate recognition models, and the recognition model is updated to the candidate recognition model; and Return to the step of using the recognition model to extract features from each training sample pair until the number of updates reaches the preset first number, and obtain the target encoding network.

10. The image recognition method according to claim 9, wherein, The step of determining the coding loss information of the coding network based on the global loss information and the local similarity set includes: The target global loss information corresponding to the target training sample pair is selected from the global loss information; Based on the local similarity set, determine the local loss information corresponding to the encoding network; and The global loss information and the local loss information are fused to obtain the coding loss information of the coding network.

11. The image recognition method according to claim 10, wherein, The step of determining the local loss information corresponding to the encoding network based on the local similarity set includes: The local similarity is traversed in the local similarity set to obtain the local similarity corresponding to each text sub-feature, thus obtaining the candidate local similarity set corresponding to each text sub-feature; The local similarity with the highest similarity is selected from the candidate local similarity set to obtain the target local similarity corresponding to each text sub-feature; and The target local similarity is adjusted to a preset similarity to obtain the local loss information corresponding to the encoding network, wherein the preset similarity is greater than the target local similarity.

12. The image recognition method according to claim 7, wherein, The step of training the sampling network based on the target encoding network by performing a preset number of samplings a second time to obtain the target sampling network includes: The target encoding network is used to extract features from the at least one training sample pair to obtain the current global loss information set corresponding to the at least one training sample pair; Based on the sampling network, candidate global loss information is selected from the current global loss information set, and target test loss information is extracted from the candidate global loss information; The network parameters of the sampling network are updated based on the target test loss information to obtain the current sampling network, and the sampling network is updated to the current sampling network; and Return to the step of using the target encoding network to extract features from the at least one training sample pair, until the number of updates reaches the preset second number, to obtain the target sampling network.

13. The image recognition method according to claim 12, wherein, The step of extracting the target test loss information from the candidate global loss information includes: From the candidate global loss information, at least one global loss information corresponding to each test sample pair is selected to obtain a test loss information set; and The minimum test loss information is selected from the set of test loss information to obtain the target test loss information.

14. The image recognition method according to claim 1, wherein, Before extracting features from each training sample pair using the recognition model, the method further includes: Using the sampling network in the preset recognition model and the at least one training sample pair, the encoding network in the preset recognition model is trained for a preset first number of encoding operations to obtain the basic encoding network. Based on the basic coding network and the at least one training sample pair, the sampling network in the preset recognition model is trained with a preset second number of samplings to obtain the basic sampling network; The basic coding network and basic sampling network are used as the basic recognition model, and the preset recognition model is updated to the basic recognition model; and Return to the step of performing a preset first number of encoding training steps on the encoding network in the preset recognition model using the sampling network in the preset recognition model and the at least one training sample pair, until the preset recognition model converges, to obtain the recognition model.

15. The image recognition method according to claim 14, wherein, The step of using the sampling network in the preset recognition model and the at least one training sample pair to perform a preset first number of encoding training operations on the encoding network in the preset recognition model to obtain the basic encoding network includes: The encoding network in the preset recognition model is used to extract features from the at least one training sample pair to obtain a set of candidate global loss information corresponding to the at least one training sample pair. Based on the sampling network in the preset recognition model, basic global loss information is selected from the candidate global loss information set; and Based on the basic global loss information, the encoding network in the preset recognition model is trained with the preset first number of encoding operations to obtain the basic encoding network.

16. The image recognition method according to claim 1, wherein, After the target recognition model is obtained by converging the recognition model based on the local similarity set, the method further includes: Obtain a text set corresponding to at least one preset attribute information of the target object; and The target recognition model is used to extract features from each text in the text set to obtain a text feature set corresponding to the at least one preset attribute information.

17. The image recognition method according to claim 16, wherein, The step of using the target recognition model to identify the attribute information of the target object in the object image includes: Obtain the object image of the target object; The target recognition model is used to extract features from the object image to obtain the current image features of the object image; and Based on the text feature set and the current image features, the attribute information of the target object in the object image is determined.

18. The image recognition method according to claim 17, wherein, The step of determining the attribute information of the target object in the object image based on the current image features from the text feature set includes: Obtain the similarity between the current image feature and each text feature in the text feature set to obtain the current similarity set; Target similarities exceeding a preset similarity threshold are selected from the current similarity set; and The attribute information corresponding to the target similarity is selected from the at least one attribute information to obtain the attribute information of the target object.

19. The image recognition method according to claim 16, wherein, Also includes: Obtain the target text corresponding to the current attribute information, and update the at least one attribute information based on the current attribute information to obtain the updated attribute information set; The target recognition model is used to extract features from the target text to obtain the current text features; as well as Based on the current text features, the text feature set is updated to obtain the updated text feature set corresponding to the attribute information set.

20. An image recognition system, comprising: At least one storage medium storing at least one instruction set for image recognition; as well as At least one processor is communicatively connected to the at least one storage medium. When the image recognition system is running, the at least one processor reads the at least one instruction set and executes the image recognition method according to any one of claims 1-19 according to the instructions of the at least one instruction set.

Images