Target recognition model training method and device, storage medium and equipment

Abstract

The invention discloses a target recognition model training method and device, a storage medium and equipment, and the method comprises the steps: obtaining an original image, determining an area corresponding to a target object in the original image as a sample image, inputting the sample image into a pre-trained multi-modal large model, determining a description text corresponding to the sample image as a sample text, and carrying out the recognition of the target object through the sample text. And inputting the original image and the sample text into the target recognition model, determining image features output by the image encoder for the sample image, determining text features output by the text encoder for the sample text, and determining the target recognition model according to the matching degree of the image features and the text features. According to the method, at least the image encoder is trained, and a high-precision target recognition model can be obtained through training without pre-establishing a strictly labeled sample set.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a training method, apparatus, storage medium, and device for a target recognition model. Background Art

[0002] Currently, training target recognition models mostly relies on a pre-built vocabulary pool. A common training method is to manually select corresponding words from the vocabulary pool and annotate them to the target area of each original image, thereby using the annotated original images to train the target recognition model. In this method, the process of annotating sample images is too dependent on manual labor, which seriously affects the efficiency of model training.

[0003] Another method is to use a pre-trained target recognition model to match the words in the vocabulary pool with the target area of each original image to determine the label corresponding to each target area. Since the pre-trained target recognition model is difficult to achieve accurate matching effects, the sample images obtained by labeling using this method contain more noise information.

[0004] The above training method is widely used in the field of zero-shot detection to train target detection models. However, the above training method seriously affects the stability and generalization performance of the trained target recognition model in actual application scenarios.

[0005] Therefore, the present invention provides a training method, device, storage medium, and equipment for a target recognition model. Summary of the Invention

[0006] This specification provides a method and device for training a target recognition model to partially solve the above-mentioned problems existing in the prior art.

[0007] This manual adopts the following technical solutions:

[0008] This specification provides a training method for an object recognition model. The object recognition model includes a text encoder and an image encoder, including:

[0009] Get the original image;

[0010] Determine an area corresponding to the target object in the original image as a sample image;

[0011] Inputting the sample image into a pre-trained multimodal large model, and determining a description text corresponding to the sample image as the sample text;

[0012] Inputting the original image and the sample text into the object recognition model, determining image features output by the image encoder for the sample image, and determining text features output by the text encoder for the sample text;

[0013] At least the image encoder is trained according to the matching degree between the image feature and the text feature.

[0014] Optionally, determining a region corresponding to the target object in the original image as a sample image specifically includes:

[0015] Determining a target area where a target object is located in the original image;

[0016] The target area in the original image is expanded and cropped to obtain a sample image corresponding to the target object.

[0017] Optionally, determining a region corresponding to the target object in the original image as a sample image specifically includes:

[0018] Determine the target area where the target object is located in the original image, and obtain a sample image corresponding to the target object after cropping;

[0019] Determining the description text corresponding to the sample image as the sample text specifically includes:

[0020] Describe the noun corresponding to the sample image as the sample text.

[0021] Optionally, determining a region corresponding to the target object in the original image as a sample image specifically includes:

[0022] Determine a sample image corresponding to each target object in the original image;

[0023] Inputting the original image and the sample text into the object recognition model, determining image features output by the image encoder for the sample image, and determining text features output by the text encoder for the sample text, specifically includes:

[0024] Inputting the original image and the sample text corresponding to each sample image into the object recognition model, determining the image features output by the image encoder for each sample image, and determining the text features output by the text encoder for each sample text;

[0025] According to the matching degree between the image feature and the text feature, at least the image encoder is trained, specifically comprising:

[0026] At least the image encoder is trained according to the matching degree between the image features and the text features corresponding to the same sample image and the matching degree between the image features and the text features corresponding to different sample images.

[0027] Optionally, the dimension of the image feature is the same as that of the text feature;

[0028] Training at least the image encoder according to the matching degree between the image features and the text features corresponding to the same sample image and the matching degree between the image features and the text features corresponding to different sample images, specifically comprising:

[0029] For any image feature, determine the matching degree between the image feature and each text feature based on the product of the image feature and each text feature;

[0030] At least the image encoder is trained according to the matching degree between the image features and the text features corresponding to the same sample image and the matching degree between the image features and the text features corresponding to different sample images.

[0031] Optionally, at least the image encoder is trained based on the matching degree between the image features and the text features corresponding to the same sample image and the matching degree between the image features and the text features corresponding to different sample images, specifically including:

[0032] Determine a loss value using a preset loss function based on the matching degree between the image features and the text features corresponding to the same sample image and the matching degree between the image features and the text features corresponding to different sample images;

[0033] At least the image encoder is trained based on the loss value.

[0034] Optionally, also include:

[0035] After the target recognition model is trained, when a target image is recognized, a target image is obtained;

[0036] determining the regions where each target object is located in the target image as a plurality of target regions;

[0037] Input the target image into the image encoder of the trained target recognition model, and determine the image code corresponding to each target object according to the target area as the code to be matched;

[0038] The category of the target object in the target image is determined according to the matching degree between the code to be matched and the text code corresponding to each target text.

[0039] This specification provides a training device for an object recognition model. The object recognition model includes a text encoder and an image encoder, including:

[0040] Acquisition module, obtains the original image;

[0041] A determination module determines an area corresponding to the target object in the original image as a sample image;

[0042] A description module inputs the sample image into a pre-trained multimodal large model and determines a description text corresponding to the sample image as a sample text;

[0043] an extraction module, inputting the original image and the sample text into the object recognition model, determining image features output by the image encoder for the sample image, and determining text features output by the text encoder for the sample text;

[0044] A training module is configured to train at least the image encoder according to a matching degree between the image feature and the text feature.

[0045] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the training method of the above-mentioned target recognition model.

[0046] This specification provides a device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the training method of the target recognition model described above is implemented.

[0047] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0048] In the training method of the target recognition model provided in this specification, an original image is obtained, and the area corresponding to the target object in the original image is determined as a sample image. The sample image is input into a pre-trained multimodal large model, and the descriptive text corresponding to the sample image is determined as sample text. The original image and the sample text are input into the target recognition model, and the image features output by the image encoder for the sample image are determined, and the text features output by the text encoder for the sample text are determined. According to the matching degree between the image features and the text features, at least the image encoder is trained.

[0049] It can be seen from the above method that this method can train a highly accurate target recognition model without pre-establishing a strictly labeled sample set. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0051] Figure 1 This is a flowchart of a method for training an object recognition model in this specification;

[0052] Figure 2This is a schematic diagram of the effect of expanding the target area in this manual;

[0053] Figure 3 A flowchart of a method for training an object recognition model for different types of sample images in this specification;

[0054] Figure 4 This is a flowchart of applying the target recognition model to perform target recognition in this manual;

[0055] Figure 5 A schematic diagram of a training device for an object recognition model provided in this specification;

[0056] Figure 6 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0057] To make the purpose, technical solutions, and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0058] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0059] Figure 1 The following is a flow chart of a method for training a target recognition model in this specification. The method for training a target recognition model specifically includes the following steps:

[0060] S100: Acquire an original image.

[0061] In various current business scenarios, target detection tasks are mostly to identify images collected in open scenes. Such images have various characteristics such as high image complexity, many targets in the image, and many types of targets in the image. Using traditional target detection models to identify such images, on the one hand, has extremely high requirements for the data set used for training. On the other hand, when there is insufficient data in the accurately labeled data set, it is difficult for the target detection model to achieve accurate target recognition. Therefore, this specification provides a training method for a target detection model. The execution subject of this specification can be a server for training a target recognition model, or other electronic devices that have the ability to connect to the target recognition model. This specification does not limit this. For the sake of convenience, the following only uses the server as the execution subject to illustrate a training method for a target recognition model provided in this specification.

[0062] In the method provided in this specification, the target recognition model includes at least an image encoder and a text encoder. The image encoder can encode the input image to obtain image features corresponding to the image, and the text encoder can encode the input text to obtain text features corresponding to the text.

[0063] First, an original image for training a target detection model is obtained. In the method provided in this specification, the original image may be an image containing one or more target objects and scene information.

[0064] The target object may be an object that is recognized by the trained target recognition model.

[0065] S102: Determine an area corresponding to the target object in the original image as a sample image.

[0066] As described above, the original image may contain several target objects. For one target object, a local image containing the target object in the original image is determined as the sample image corresponding to the target object.

[0067] The shape of the partial image can be set by the user.

[0068] S104: Input the sample image into a pre-trained multimodal large model, and determine the description text corresponding to the sample image as the sample text.

[0069] After obtaining the sample image, the sample image can be input into a pre-trained multimodal large model, and the multimodal large model can generate a description text corresponding to the sample image as the sample text based on the input sample image.

[0070] In one or more embodiments of the present specification, a preset command statement and the sample image may be input into the multimodal large model, causing the multimodal large model to output a description text in a unified format. Specifically, the description text may include a description of the overall scene of the sample image and a description of the target object contained in the sample image.

[0071] Among them, the above-mentioned multimodal large model can be selected from common multimodal large models such as BLIP2 and Llava, and this manual does not limit it here.

[0072] S106: Input the original image and the sample text into the target recognition model, determine the image features output by the image encoder for the sample image, and determine the text features output by the text encoder for the sample text.

[0073] After obtaining the sample text corresponding to the sample image, the original image and the sample text can be input into the target recognition model to determine the image features obtained by the image encoder in the target recognition model based on the sample image, and the text features obtained by the text encoder in the target recognition model based on the sample text.

[0074] The image encoder and the text encoder can capture key features in an image or text and convert them into vector form that can be used for subsequent calculations. Specifically, the above-mentioned image encoder and text encoder can adopt common encoder architectures, such as Transformer architecture, convolutional neural network, etc., and this specification does not impose any restrictions on this.

[0075] Among them, after the original image is input into the image encoder, the image encoder can generate a first image feature corresponding to the original image, and the first image feature can be expressed in a matrix form. Furthermore, according to the positional relationship between the sample image and the original image, the corresponding area of the sample image is intercepted in the first image feature according to the positional relationship as the second image feature corresponding to the sample image. Finally, the second image feature is standardized to a pre-set size to obtain the final image feature corresponding to the sample image.

[0076] Specifically, the second image feature may be normalized to a preset size by using the RoI Align method.

[0077] S108: Training at least the image encoder according to the matching degree between the image feature and the text feature.

[0078] After obtaining the image features of the sample image and the text features of the sample text corresponding to the sample image, the matching degree between the image features and the text features can be determined, and the target recognition model is trained based on the matching degree between the image features and the text features.

[0079] Among them, to determine the matching degree between image features and text features, various common similarity calculation formulas, such as cosine similarity, Euclidean distance, etc., can be used to regard the similarity between image features and text features as the matching degree; the target recognition model can be trained to maximize the matching degree between the image features and text features corresponding to the same sample image, and at least the parameters of the image encoder in the target recognition model can be adjusted.

[0080] like Figure 1 The training method of a target recognition model shown in the figure can train a highly accurate target recognition model without pre-establishing a strictly labeled sample set.

[0081] In addition, in Figure 1In step S102 shown, a target region where the target object is located in the original image is determined, the target region in the original image is expanded, and a sample image corresponding to the target object is obtained after cropping.

[0082] Specifically, a pre-trained target detection model can be used to determine the target area where the target object in the original image is located, and the size of the target area can be expanded to obtain an expanded target area that includes the original target area but has a larger range. The expanded target area is mapped to the original image through coordinate mapping, and the mapped area in the original image is cropped to obtain a sample image.

[0083] Among them, the target object can be the recognition target of the target recognition model trained by the user, and based on different application scenarios, it can include different types of objects, for example; for an original image, several target areas can be determined in the original image, and correspondingly, one original image can correspond to several sample images; the above-mentioned target detection model can adopt various common model architectures, such as EfficientDet, FasterRCNN (Faster Region-based Convolutional NeuralNetworks), YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), FCOS (Fully Convolutional One-Stage), DETR (DEtection TRansformer), etc., and this manual does not limit this.

[0084] like Figure 2 As shown, Figure 2 The dotted box in the middle represents the target area. According to the distribution law of semantic information in the image, several target objects that are different from the background in an image usually have concentrated distribution characteristics. The expanded target area usually has more target objects and rich semantic information. The sample image obtained by cropping the expanded target area is used for subsequent training of the target recognition model. The trained target recognition model can learn regional-level recognition capabilities. When using the target recognition model for image recognition, the semantic information corresponding to the image containing complex information (which may be a combination of multiple target objects to form a specific scene) can be obtained.

[0085] It should be noted that, when the sample image is obtained by expanding and cropping the target area in the original image, the description text corresponding to the sample image may be a number of descriptive nouns or a description sentence indicating the type of the target object.

[0086] On the other hand, in Figure 1In step S102 shown in FIG. 1 , the target area where the target object is located in the original image is determined, and a sample image corresponding to the target object is obtained after cropping; Figure 1 In step S104 shown, the description noun corresponding to the sample image is determined as the sample text.

[0087] It is also possible to determine a target area that only contains a unique target object, determine the mapping of the target area in the sample image, and use it as a sample image after cropping. The descriptive text of the sample image can be the descriptive noun corresponding to the target object. The subsequent target recognition model is trained based on the sample image obtained by direct cropping. The trained target recognition model can learn instance-level (for sample images containing only a single target object) recognition capabilities. When using the target recognition model for image recognition, more accurate semantic information of images containing simple information (containing only a single target object) can be obtained.

[0088] Figure 3 This is a flowchart of a method for training an object recognition model for different types of sample images in this specification. First, step S300 is performed: obtaining an original image.

[0089] Step S302 is then executed: a region corresponding to the target object in the original image is determined as a sample image.

[0090] If the sample image is obtained by expanding and cropping the target area in the original image, execute step S304A: input the sample image into a pre-trained multimodal large model, and determine the description text corresponding to the sample image as the sample text. The description text may include several descriptive nouns or a descriptive sentence; if the sample image is directly obtained after cropping the target area where the target object in the original image is located, execute step S304B: input the sample image into a pre-trained multimodal large model, and determine the description noun corresponding to the sample image as the sample text.

[0091] Then, step S306 is performed: the original image and the sample text are input into the target recognition model, the image features output by the image encoder for the sample image are determined, and the text features output by the text encoder for the sample text are determined.

[0092] Finally, step S308 is performed: at least the image encoder is trained according to the matching degree between the image feature and the text feature.

[0093] In addition, in Figure 1 In step S102 shown in FIG. 1 , a sample image corresponding to each target object in the original image is determined; Figure 1In step S106 shown in FIG, the original image and the sample text corresponding to each sample image are input into the target recognition model, the image features output by the image encoder for each sample image are determined, and the text features output by the text encoder for each sample text are determined; Figure 1 In step S108 shown, at least the image encoder is trained based on the matching degree between the image features and the text features corresponding to the same sample image and the matching degree between the image features and the text features corresponding to different sample images.

[0094] When training the target recognition model based on sample images, a comparative learning method can be adopted. Specifically, several sample images can be used for training, so that for each sample image, each sample image has a corresponding sample text describing the sample image. Then, the image encoder of the target recognition model determines the image features output for each sample image, and the text encoder determines the text features output for each sample text. According to the matching degree between the image features and text features corresponding to the same sample image and the matching degree between the image features and text features corresponding to different sample images, at least the image encoder is trained.

[0095] Specifically, the target recognition model can be trained with the goal of improving the matching degree between image features and text features corresponding to the same sample image, and the matching degree between image features and text features corresponding to different sample images; the target recognition model can also be trained with the goal of improving the matching degree between image features and text features corresponding to the same sample image, and reducing the similarity between positive samples and negative sample pairs, or other feasible training methods are not limited in this specification. Among them, the positive sample pair is a pair of image features and text features corresponding to the same sample image, and the negative sample pair is a pair of image features and text features corresponding to different sample images.

[0096] In this way, multiple groups of sample images can be used simultaneously to improve the efficiency and performance of training the target recognition model.

[0097] Furthermore, in Figure 1 In step S106 shown, a loss value is determined based on the matching degree between the image features and text features corresponding to the same sample image and the matching degree between the image features and text features corresponding to different sample images using a preset loss function, and at least the image encoder is trained based on the loss value.

[0098] According to the user's application requirements, different loss functions can be preset, such as InfoNCE Loss in contrastive loss (ContrastiveLoss) or other commonly used loss functions. The image features and text features are substituted into the preset loss function to determine the loss value, and the gradient of the loss relative to the parameters of each model layer in the target recognition model is determined. According to the preset learning rate, the parameters of each model layer of the target recognition model are adjusted.

[0099] In addition, in one or more embodiments of the present specification, after the target recognition model is trained, when a target image is recognized, a target image is obtained, and the target image is cropped according to the area where each target object in the target image is located to obtain several sub-target images. For any sub-target image, the sub-target image is input into the image encoder of the trained target recognition model, and the image encoding corresponding to the sub-target image is determined as the encoding to be matched. According to the matching degree between the encoding to be matched and the text encoding corresponding to each target text, the category of the target object in the sub-target image is determined.

[0100] Specifically, such as Figure 4 As shown, first, step S400 is performed: acquiring a target image.

[0101] Then, step S402 is executed: determining the regions where each target object in the target image is located as a plurality of target regions.

[0102] Then, step S404 is executed: the target image is input into the image encoder of the trained target recognition model to determine the image code corresponding to each target object as the code to be matched.

[0103] Finally, step S406 is executed: determining the category of the target object in the target image according to the matching degree between the code to be matched and the text code corresponding to each target text.

[0104] Among them, the target image obtained can be the original image collected in the actual application scenario, and the pre-trained target detection model can be used to determine the target area in the target image; the target text can be set by the user according to the actual application scenario. For example, in the h5 (APP side or mobile web page side) inspection scenario, when conducting APP review business, the trained target recognition model can be used to identify the image obtained by the screenshot, and the target object in the image - the business function entrance can be determined. The request area corresponding to different business functions in the APP can be located, and the test level for different business functions can be used to automatically test each business function in the APP. In this scenario, each business function in the APP can be set as the target text.

[0105] The above is a training method for a target recognition model provided in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding training device for a target recognition model, such as Figure 5 shown.

[0106] Figure 5 This is a schematic diagram of a training device for an object recognition model provided in this specification. The object recognition model includes a text encoder and an image encoder. The device specifically includes:

[0107] An acquisition module 500 acquires an original image;

[0108] Determination module 502, determines the area corresponding to the target object in the original image as a sample image

[0109] The description module 504 inputs the sample image into a pre-trained multimodal large model and determines a description text corresponding to the sample image as the sample text;

[0110] The extraction module 506 inputs the original image and the sample text into the object recognition model, determines the image features output by the image encoder for the sample image, and determines the text features output by the text encoder for the sample text;

[0111] The training module 508 is configured to train at least the image encoder according to the matching degree between the image feature and the text feature.

[0112] Optionally, the determination module 502 is specifically configured to: determine a target region where the target object is located in the original image, expand the target region in the original image, and obtain a sample image corresponding to the target object after cropping.

[0113] Optionally, the determination module 502 is specifically configured to: determine a target area where a target object is located in the original image, and obtain a sample image corresponding to the target object after cropping;

[0114] The description module 504 is specifically configured to determine a description noun corresponding to the sample image as a sample text.

[0115] Optionally, the determination module 502 is specifically configured to: determine a sample image corresponding to each target object in the original image;

[0116] The extraction module 506 is specifically configured to: input the original image and the sample text corresponding to each sample image into the object recognition model, determine the image features output by the image encoder for each sample image, and determine the text features output by the text encoder for each sample text;

[0117] The training module 508 is specifically configured to train at least the image encoder according to the matching degree between the image features and the text features corresponding to the same sample image and the matching degree between the image features and the text features corresponding to different sample images.

[0118] Optionally, the dimension of the image feature is the same as that of the text feature;

[0119] The training module 508 is specifically used to: for any image feature, determine the matching degree between the image feature and each text feature based on the product of the image feature and each text feature, and train at least the image encoder based on the matching degree between the image feature and the text feature corresponding to the same sample image and the matching degree between the image feature and the text feature corresponding to different sample images.

[0120] Optionally, the training module 508 is specifically used to: determine the loss value based on the matching degree between the image features and text features corresponding to the same sample image and the matching degree between the image features and text features corresponding to different sample images using a preset loss function, and train at least the image encoder based on the loss value.

[0121] Optionally, the training module 508 is also used to: after the target recognition model is trained, when the target image is recognized, obtain the target image, determine the areas where each target object in the target image is located as several target areas, input the target image into the image encoder of the trained target recognition model, and determine the image code corresponding to each target object according to the target area as the code to be matched, and determine the category of the target object in the target image according to the matching degree between the code to be matched and the text code corresponding to each target text.

[0122] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 Provides training methods for object recognition models.

[0123] This manual also provides Figure 6 The schematic structure diagram of the electronic device shown in FIG. Figure 6 As mentioned above, at the hardware level, the training equipment of the target recognition model includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1The training method of the target recognition model described above. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0124] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0125] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0126] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0127] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0128] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0132] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0133] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0134] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0135] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0136] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0138] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0139] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be encompassed within the scope of the claims of this application.

Claims

1. A method for training an object recognition model, wherein the object recognition model includes a text encoder and an image encoder, the method comprising: Get the original image; Determine an area corresponding to the target object in the original image as a sample image; Inputting the sample image into a pre-trained multimodal large model, and determining a description text corresponding to the sample image as the sample text; Inputting the original image and the sample text into the object recognition model, determining image features output by the image encoder for the sample image, and determining text features output by the text encoder for the sample text; At least the image encoder is trained according to the matching degree between the image feature and the text feature.

2. The method of claim 1, wherein determining the area corresponding to the target object in the original image as the sample image comprises: Determining a target area where a target object is located in the original image; The target area in the original image is expanded and cropped to obtain a sample image corresponding to the target object.

3. The method of claim 1, wherein determining the area corresponding to the target object in the original image as the sample image comprises: Determine the target area where the target object is located in the original image, and obtain a sample image corresponding to the target object after cropping; Determining the description text corresponding to the sample image as the sample text specifically includes: Describe the noun corresponding to the sample image as the sample text.

4. The method of claim 1, wherein determining the area corresponding to the target object in the original image as the sample image comprises: Determine a sample image corresponding to each target object in the original image; Inputting the original image and the sample text into the object recognition model, determining image features output by the image encoder for the sample image, and determining text features output by the text encoder for the sample text, specifically includes: Inputting the original image and the sample text corresponding to each sample image into the object recognition model, determining the image features output by the image encoder for each sample image, and determining the text features output by the text encoder for each sample text; According to the matching degree between the image feature and the text feature, at least the image encoder is trained, specifically comprising: At least the image encoder is trained according to the matching degree between the image features and the text features corresponding to the same sample image and the matching degree between the image features and the text features corresponding to different sample images.

5. The method of claim 1, wherein the image feature and the text feature have the same dimension; Training at least the image encoder according to the matching degree between the image features and the text features corresponding to the same sample image and the matching degree between the image features and the text features corresponding to different sample images, specifically comprising: For any image feature, determine the matching degree between the image feature and each text feature based on the product of the image feature and each text feature; At least the image encoder is trained according to the matching degree between the image features and the text features corresponding to the same sample image and the matching degree between the image features and the text features corresponding to different sample images.

6. The method of claim 5, wherein at least the image encoder is trained based on the matching degree between the image features and the text features corresponding to the same sample image and the matching degree between the image features and the text features corresponding to different sample images, specifically comprising: Determine a loss value using a preset loss function based on the matching degree between the image features and the text features corresponding to the same sample image and the matching degree between the image features and the text features corresponding to different sample images; At least the image encoder is trained based on the loss value.

7. The method of claim 1 , further comprising: After the target recognition model is trained, when a target image is recognized, a target image is obtained; determining the regions where each target object is located in the target image as a plurality of target regions; Input the target image into the image encoder of the trained target recognition model, and determine the image code corresponding to each target object according to the target area as the code to be matched; The category of the target object in the target image is determined according to the matching degree between the code to be matched and the text code corresponding to each target text.

8. A training device for an object recognition model, the object recognition model including a text encoder and an image encoder, the device comprising: Acquisition module, obtains the original image; A determination module determines an area corresponding to the target object in the original image as a sample image; A description module inputs the sample image into a pre-trained multimodal large model and determines a description text corresponding to the sample image as a sample text; an extraction module, inputting the original image and the sample text into the object recognition model, determining image features output by the image encoder for the sample image, and determining text features output by the text encoder for the sample text; A training module is configured to train at least the image encoder according to a matching degree between the image feature and the text feature.

9. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 7.

10. A device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

Images