Target recognition method, device, system, electronic device and storage medium

Abstract

The present application discloses a target recognition method, device, system, and electronic device, belonging to the field of artificial intelligence technology. The method includes: presetting a target recognition model including an image feature extraction module, a language feature extraction module, and a target extraction module, where the image feature extraction module includes a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, and the target extraction module includes a second hybrid adapter multi-expert module composed of a plurality of second expert modules; obtaining an image to be recognized and a prompt text in a target scenario; automatically adapting the first expert module to extract the image features of the image to be recognized; extracting the language features of the prompt text through the language feature extraction module; automatically adapting the second expert module to extract the multi-modal features of the image features and the language features, and obtaining the target recognition result corresponding to the image to be recognized based on the multi-modal features. This method can achieve multiple target recognitions through one model, shortening the R & D cycle of related applications.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular, to an object recognition method, an object recognition device, an object recognition system, an electronic device, and a computer-readable storage medium. Background Art

[0002] A "long-tail object" refers to an object with a relatively low occurrence probability, including but not limited to: events, behaviors, people, animals, etc. With the improvement of the hardware performance of security cameras, the increase in resolution, and the development of artificial intelligence technology, based on security cameras and artificial intelligence algorithms, various objects in cities and rural areas can be recognized to ensure the increasingly widespread application of social security. For example, surveillance site images can be collected through security cameras, and artificial intelligence technology can be used to identify whether there are objects that violate social security or urban environmental requirements in the collected images for subsequent warning, public security management, and other applications.

[0003] However, in the prior art, the recognition requirements for long-tail objects are diverse and tend to be customized and fragmented. For the diverse recognition requirements, where the long-tail objects are different, when the recognition algorithm based on traditional artificial intelligence methods recognizes long-tail objects in a specified scenario, customized development needs to be carried out for the recognition requirements of these long-tail objects, resulting in defects such as difficult data collection, long R & D cycles, high R & D costs, and high costs for manually optimizing algorithms.

[0004] It can be seen that the object recognition method in the prior art still needs to be improved. Summary of the Invention

[0005] Embodiments of the present application provide an object recognition method and device, which help to shorten the R & D cycle of object recognition applications and reduce R & D and maintenance costs.

[0006] In a first aspect, embodiments of the present application provide an object recognition method, including:

[0007] Presetting an object recognition model, where the object recognition model includes: an image feature extraction module, a language feature extraction module, and an object extraction module. Among them, the image feature extraction module includes: a first hybrid adapter multi-expert module composed of multiple different first expert modules, and the object extraction module includes: a second hybrid adapter multi-expert module composed of multiple different second expert modules, and a first output module;

[0008] Obtaining an image to be recognized in a target scenario and a prompt text indicating the object to be recognized;

[0009] Performing image feature extraction on the image to be recognized through one or more of the first expert modules automatically adapted to the target scenario to obtain image features; and performing language feature extraction on the prompt text through the language feature extraction module to obtain language features;

[0010] Performing multi-modal feature extraction on the image features and the language features through one or more of the second expert modules automatically adapted to the target scenario to obtain multi-modal features;

[0011] Obtaining the target recognition result corresponding to the image to be recognized by performing feature mapping on the multi-modal features through the first output module.

[0012] In a second aspect, an embodiment of the present application provides a target recognition device, including:

[0013] A target recognition model setting unit, configured to preset a target recognition model, where the target recognition model includes: an image feature extraction module, a language feature extraction module, and a target extraction module. Among them, the image feature extraction module includes: a first hybrid adapter multi-expert module composed of multiple different first expert modules, and the target extraction module includes: a second hybrid adapter multi-expert module composed of multiple different second expert modules, and a first output module;

[0014] An input acquisition unit, configured to acquire an image to be recognized in a target scenario and a prompt text indicating the target to be recognized;

[0015] A first feature extraction unit, configured to perform image feature extraction on the image to be recognized through one or more of the first expert modules automatically adapted to the target scenario to obtain image features; and perform language feature extraction on the prompt text through the language feature extraction module to obtain language features;

[0016] A second feature extraction unit, configured to perform multi-modal feature extraction on the image features and the language features through one or more of the second expert modules automatically adapted to the target scenario to obtain multi-modal features;

[0017] A target recognition unit, configured to perform feature mapping on the multi-modal features through the first output module to obtain the target recognition result corresponding to the image to be recognized.

[0018] In a third aspect, an embodiment of the present application provides a target recognition system, the system includes: a preset target recognition model, wherein, the target recognition model includes: an image feature extraction module, a language feature extraction module, and a target extraction module, wherein, the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, a first output module, and the system further includes:

[0019] An image acquisition unit, configured to acquire an image to be recognized in a target scenario;

[0020] A text acquisition unit, configured to acquire a prompt text in the target scenario;

[0021] A target recognition result acquisition unit, configured to use the image to be recognized and the prompt text as inputs to the target recognition model, call the target recognition model to perform the following target recognition steps, and acquire a target recognition result corresponding to the image to be recognized: extract image features from the image to be recognized through one or more of the first expert modules automatically adapted to the target scenario to obtain image features; and, extract language features from the prompt text through the language feature extraction module to obtain language features; perform multi-modal feature extraction on the image features and the language features through one or more of the second expert modules automatically adapted to the target scenario to obtain multi-modal features; and obtain a target recognition result corresponding to the image to be recognized by performing feature mapping on the multi-modal features through the first output module.

[0022] In a fourth aspect, an embodiment of the present application further discloses an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the target recognition method described in the embodiment of the present application when executing the computer program.

[0023] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the target recognition method disclosed in the embodiment of the present application are implemented.

[0024] The object recognition method disclosed in the embodiments of the present application uses a preset object recognition model, which includes an image feature extraction module, a language feature extraction module, and an object extraction module. Among them, the image feature extraction module includes a first hybrid adapter multi-expert module composed of multiple different first expert modules, and the object extraction module includes a second hybrid adapter multi-expert module composed of multiple different second expert modules, and a first output module. Obtain an image to be recognized in the target scenario and a prompt text indicating the target to be recognized. Extract image features from the image to be recognized through one or more of the first expert modules automatically adapted to the target scenario to obtain image features. And extract language features from the prompt text through the language feature extraction module to obtain language features. Perform multi-modal feature extraction on the image features and the language features through one or more of the second expert modules automatically adapted to the target scenario to obtain multi-modal features. Perform feature mapping on the multi-modal features through the first output module to obtain the object recognition result corresponding to the image to be recognized. Through the automatic selection of the large model adapter, the problem of selecting different adapters according to different scenarios is solved. There is no need to train and deploy an object recognition model separately for the object detection requirements of each target scenario, which can greatly shorten the R & D cycle of the object recognition application and reduce the R & D and maintenance costs. By using the object recognition method disclosed in the embodiments of the present application, multiple requirements can be quickly launched based on one object recognition model, thus solving the problems of fragmented requirements, complex algorithm maintenance, and high costs.

[0025] The above description is only an overview of the technical solution of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other objects, features, and advantages of the present application more obvious and understandable, the specific embodiments of the present application are given below. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0027] Figure 1 is one of the flowcharts of the object recognition method disclosed in the embodiments of the present application;

[0028] Figure 2 is one of the structural diagrams of the object recognition model disclosed in the embodiments of the present application;

[0029] Figure 3 It is the second structural schematic diagram of the target recognition model disclosed in the embodiments of the present application;

[0030] Figure 4 It is the structural schematic diagram of the first hybrid adapter multi-expert module;

[0031] Figure 5 It is the second flowchart of the target recognition method disclosed in the embodiments of the present application;

[0032] Figure 6 It is the structural schematic diagram of the target recognition system disclosed in the embodiments of the present application;

[0033] Figure 7 It is the structural schematic diagram of the target recognition device disclosed in the embodiments of the present application;

[0034] Figure 8 Schematically shows a block diagram of an electronic device for executing the method according to the present application; and

[0035] Figure 9 Schematically shows a storage unit for holding or carrying program code for implementing the method according to the present application. Detailed implementation manners

[0036] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0037] Refer to Figure 1 , a target recognition method disclosed in the embodiments of the present application includes: step 110 to step 150.

[0038] Step 110, preset a target recognition model, the target recognition model includes: an image feature extraction module, a language feature extraction module, and a target extraction module, wherein, the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, and the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, and a first output module.

[0039] As Figure 2 shown, the target recognition model at least includes: an image feature extraction module 210, a language feature extraction module 220, and a target extraction module 230, wherein, the outputs of the image feature extraction module 210 and the language feature extraction module 220 are jointly used as the input of the target extraction module 230.

[0040] Among them, the image feature extraction module 210 is used to extract the image features of the input image, the language feature extraction module 220 is used to extract the language features of the input text, and the target extraction module 230 is used to perform target recognition based on the input image features and language features.

[0041] In some alternative embodiments, such as Figure 3 shown, the target recognition model further includes: a classification module 240, and the outputs of the image feature extraction module 210 and the language feature extraction module 220 are jointly used as the input of the classification module 240, and the classification module 240 is used to perform target classification based on the input image features and language features.

[0042] Among them, the image feature extraction module 210 is constructed based on the VIT (Vision Transformer) model. During operation, the image feature extraction module 210 divides the input image into a sequence of image patches, and performs feature extraction based on the sequence of image patches, and the output is low-level visual features (texture, color, etc.) and high-level semantic features.

[0043] In some alternative embodiments, the image feature extraction module 210 includes a plurality of transformers (transformer structures), and the first hybrid adapter multi-expert module and the first LoRA module are located in the high-level transformer structure. Setting the first hybrid adapter multi-expert module and the first LoRA module in the high-level transformer structure can support fine-tuning the target recognition model and incremental learning while avoiding catastrophic forgetting.

[0044] In some alternative embodiments, the first hybrid adapter multi-expert module includes a plurality of different first expert modules. The first hybrid adapter multi-expert module further includes: a first routing module, and the first expert modules are connected in parallel through the first routing module. Refer to Figure 4 the structure of the first hybrid adapter multi-expert module shown, the first expert module can be implemented as an FFN, the first routing module can adopt a softmax structure, and the first expert module can be connected in parallel through the first routing module in a residual manner. The first routing module is used to control the invocation of the first expert module.

[0045] Taking Figure 4Taking the structure of the first hybrid adapter multi-expert module shown as an example, the first expert module corresponding to each FFN (Feed-Forward Neural Network) can be used to perform feature processing with different functions. The first routing module with a softmax structure is used to control the specific invocation of several FFNs according to the task corresponding to the current scenario, and fuse the features output by the invoked FFNs as the output of the transformer.

[0046] The role of the FFN is to perform a fully connected transformation on the vector after position encoding or decoding, strengthening the representation of position information and the extraction of features. By increasing the depth and width of the feed-forward neural network, the first hybrid adapter multi-expert module 210 can better capture the global dependencies of the input sequence, thereby improving the accuracy of feature extraction.

[0047] In some alternative embodiments, the first LoRA module is located on the Q and V matrices of the self-attention mechanism of the transformer. When training the image feature extraction module 210, the first LoRA module is connected to Q and V in a residual manner and added to the original Q and V outputs as the outputs of Q and V. During inference, the first LoRA module is directly added to the original Q and V matrices as the outputs of Q and V.

[0048] In some alternative embodiments, the language feature extraction module can be constructed based on the bert model. The language feature extraction module is used to extract the language features of the input text. Among them, the text input to the language feature extraction module can be the description text of the recognition target. The specific implementation of the language feature extraction module refers to the prior art and will not be elaborated in the embodiments of the present application.

[0049] Optionally, the target extraction module 230 includes: a second hybrid adapter multi-expert module and a first output module. The target extraction module 230 includes multiple transformer structures, and the second hybrid adapter multi-expert module is located in each of the transformer structures.

[0050] In some alternative embodiments, the second hybrid adapter multi-expert module includes multiple different second expert modules. The second hybrid adapter multi-expert module also includes: a second routing module, and the second expert modules are connected in parallel through the second routing module. The structure of the second hybrid adapter multi-expert module refers to the first hybrid adapter multi-expert module and will not be elaborated here.

[0051] The second expert module can be implemented as an FFN, the second routing module can adopt a softmax structure, and the second expert module can be connected in parallel with the second routing module in a residual manner. The second routing module is used to control the invocation of the second expert module. Each second expert module corresponding to the FFN structure can be used to perform feature processing with different functions, and the second routing module with a softmax structure is used to control the specific invocation of several FFNs according to the task corresponding to the current scenario, and fuse the features output by the invoked FFNs to obtain multi-modal features as the output of the transformer.

[0052] Optionally, the first output module is constructed based on a fully connected layer. The multi-modal features obtained after fusion are input into the first output module for feature mapping to obtain a target recognition result. For example, the first output module includes two fully connected layers, and the multi-modal features are respectively used as the inputs of the two fully connected layers, and the two fully connected layers respectively output the positioning box of the target and the confidence score of the positioning result.

[0053] In some alternative embodiments, the target extraction module 230 further includes: a second LoRA module, and the second LoRA module is located on the Q and V matrices of the self-attention mechanism of each transformer. The combination manner of the second LoRA module with the Q and V matrices can be referred to the foregoing, and will not be elaborated here.

[0054] The target extraction module 230 stacks multiple transformers and performs fusion processing on the image features and language features through the attention mechanism at different levels, and finally outputs corresponding features.

[0055] In some alternative embodiments, the target recognition model is trained by the following method: performing a classification task in a training manner of image-text matching to train the image feature extraction module and the language feature extraction module; freezing the model parameters of the trained image feature extraction module and the language feature extraction module, and performing a multi-modal object detection task to train the target extraction module to obtain a target recognition model.

[0056] In the embodiments of the present application, the target recognition model is trained in a two-stage training manner.

[0057] In the first stage, by training a classification ability model, in a training manner of image-text matching, the image feature extraction module and the language feature extraction module are trained.

[0058] Optionally, an image-text matching training method is adopted to train the image feature extraction module and the language feature extraction module, including: performing an image classification task on the first training dataset by using the image-text matching training method to train the image feature extraction module and the language feature extraction module.

[0059] First, image-text data widely collected from the network and public datasets, with the data volume in the order of billions. Most of the objects, behaviors, etc. that have appeared in human society are included in these data. For example, category description texts are set for the pictures, and the first training dataset is constructed based on the obtained data.

[0060] After that, the image feature extraction module and the language feature extraction module are respectively used to extract features from the images and description texts in the first training dataset, the cosine similarity between the features output by the image feature extraction module and the features output by the language feature extraction module is calculated, and the image feature extraction module and the language feature extraction module are trained by training the image-text matching ability.

[0061] The image-text matching ability of the target recognition model represents the similarity between the image and the category description text. That is, if image A matches the description text T, it can be considered that the object in image A belongs to the target category corresponding to the description text T. That is, the process of training the image feature extraction module and the language feature extraction module also trains the image classification ability of the target recognition model.

[0062] For the specific implementation of training the image feature extraction module and the language feature extraction module by using the image-text matching training method, reference can be made to the training method of the clip model in the prior art, which will not be elaborated here.

[0063] In the second stage, the target extraction module is trained.

[0064] Optionally, by widely collecting public target recognition datasets such as coco, grounding datasets, etc. (training datasets for computer vision and natural language models), they are converted into the data form of description text + localization box to construct the second training dataset.

[0065] Afterwards, the model parameters of the trained image feature extraction module and the language feature extraction module are frozen, and the model parameters of the target extraction module are trained. The training process is as follows: the image feature extraction module and the language feature extraction module respectively perform image feature extraction and language feature extraction on the training samples in the second training data set; then, the extracted image features and language features are input into the target extraction module for multimodal feature extraction and feature fusion mapping to obtain the category and positioning box of the target in the training sample; then, based on the loss between the category description text and target positioning box annotated on the training sample and the category and positioning box output by the target extraction module, the model parameters of the target extraction module are fine-tuned, thereby iteratively training the target extraction module.

[0066] By combining a two-stage training method of image-text matching training and multimodal target extraction, the target recognition model has both image classification and target detection and recognition capabilities.

[0067] For the pre-trained object recognition model, when it is deployed for inference, when an image and a prompt text describing the object to be identified are input, the object recognition model can infer the positioning box of the object in the image.

[0068] Step 120: Acquire an image to be recognized in a target scene and a prompt text indicating a target to be recognized.

[0069] The target scenarios include but are not limited to: designated behavior recognition (such as driving against traffic, littering, etc.), large dog recognition, etc.

[0070] The image to be identified in the target scene can be acquired using image acquisition technology. The specific implementation method for acquiring the image to be identified in the target scene is not limited in the embodiments of this application. The prompt text is set according to the target identification requirements. Taking the scenario of identifying wrong-way traffic as an example, the prompt text can be "Wrong-way traffic vehicle".

[0071] Step 130 , performing image feature extraction on the image to be identified to obtain image features by using one or more of the first expert modules that are automatically adapted to the target scene; and performing language feature extraction on the prompt text to obtain language features by using the language feature extraction module.

[0072] As described above, the image feature extraction module includes: a first hybrid adapter multi-expert module, where the first hybrid adapter multi-expert module further includes: a plurality of the first expert modules and a first routing module. Correspondingly, in some alternative embodiments, extracting image features from the image to be recognized by one or more of the first expert modules that are automatically adapted to the target scenario to obtain image features includes: calling one or more of the first expert modules associated with the target scenario through the first routing module to obtain the single-expert module output features output by the called first expert modules; fusing the single-expert module output features to obtain image features.

[0073] Optionally, one or more of the first expert modules associated with the target scenario can be preset according to the test results of the target recognition model. For example, the first expert modules to be called under different target scenarios can be preset.

[0074] In some other alternative embodiments, one or more of the first expert modules associated with the target scenario can be determined according to the call weights of the first expert modules obtained from training the target recognition model. For example, when training the target recognition model based on the training data of different target scenarios, record the weights of each first expert module in the target recognition model for each target scenario. During the target recognition phase, the first routing module calls the corresponding first expert module according to the weights determined in the training phase.

[0075] During specific implementation, after inputting the image to be recognized into the image feature extraction module, the first routing module in the image feature extraction module automatically selects one or more of the first expert modules to perform feature extraction on the image to be recognized respectively and outputs the corresponding image features. Then, fuse the image features output by the selected first expert modules as the image features output by the image feature extraction module.

[0076] On the other hand, input the prompt text into the language feature extraction module. After the language feature extraction module extracts the language features of the prompt text, it outputs the language features of the prompt text.

[0077] Step 140, perform multi-modal feature extraction on the image features and the language features through one or more of the second expert modules that are automatically adapted to the target scenario to obtain multi-modal features.

[0078] Next, the image features output by the image feature extraction module and the language features output by the language feature extraction module are input into the target extraction module.

[0079] As described above, the target extraction module includes: the second hybrid adapter multi-expert module and the first output module, where the second hybrid adapter multi-expert module includes: a plurality of different second expert modules and a second routing module, and the second expert modules are connected in parallel through the second routing module. Correspondingly, one or more of the second expert modules that are automatically adapted to the target scenario perform multi-modal feature extraction on the image features and the language features to obtain multi-modal features, including: calling, through the second routing module, one or more of the second expert modules associated with the target scenario to perform multi-modal feature extraction on the image features and the language features to obtain the multi-modal output of a single expert module; and fusing the multi-modal output of the single expert module to obtain multi-modal features.

[0080] Among them, for the determination method of one or more of the second expert modules associated with the target scenario, refer to the determination method of one or more of the first expert modules associated with the target scenario described above, which will not be elaborated here.

[0081] In specific implementation, after inputting the image features and the language features into the target extraction module, the second routing module in the target extraction module automatically selects one or more of the second expert modules to respectively perform feature processing on the image features and the language features, and outputs the corresponding feature processing results as the multi-modal output of a single expert module. Then, the second routing module fuses the multi-modal outputs of each single expert module to obtain a fused feature as the multi-modal feature extracted from the image features and the language features.

[0082] Step 150, perform feature mapping on the multi-modal features through the first output module to obtain the target recognition result corresponding to the image to be recognized.

[0083] Optionally, the first output module includes two fully connected layers. One fully connected layer is used to perform feature mapping processing on the multi-modal features to obtain the localization box of the target to be recognized; the other fully connected layer is used to perform feature mapping processing on the multi-modal features to obtain the confidence score of the localization result corresponding to the localization box. Further, according to the confidence score of the localization result, the localization box of the target to be recognized can be determined. For example, select the localization box with the highest confidence score as the localization box of the target to be recognized.

[0084] Take the localization box and the confidence as the target recognition result.

[0085] Further, according to the target to be recognized indicated by the prompt text, it can be determined whether the current image to be recognized is an alarm image. That is, if an alarm target is recognized in the image to be recognized, the image to be recognized is an alarm image.

[0086] In summary, the target recognition method disclosed in the embodiments of the present application, through a preset target recognition model, the target recognition model includes: an image feature extraction module, a language feature extraction module, and a target extraction module. Among them, the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, and the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, and a first output module; obtaining an image to be recognized in a target scenario and a prompt text indicating the target to be recognized; extracting image features from the image to be recognized through one or more of the first expert modules automatically adapted to the target scenario to obtain image features; and extracting language features from the prompt text through the language feature extraction module to obtain language features; extracting multi-modal features from the image features and the language features through one or more of the second expert modules automatically adapted to the target scenario to obtain multi-modal features; obtaining a target recognition result corresponding to the image to be recognized by performing feature mapping on the multi-modal features through the first output module. Through the automatic selection of the large model adapter, the problem of needing to select different adapters according to different scenarios is solved. There is no need to train and deploy a target recognition model separately for the target detection requirements of each target scenario, which can greatly shorten the R & D cycle of the target recognition application and reduce the R & D and maintenance costs. By using the target recognition method disclosed in the embodiments of the present application, multiple requirements can be quickly launched based on a single target recognition model, thus solving the problems of fragmented requirements, complex algorithm maintenance, and high costs.

[0087] For example, after training the target recognition model, for a specific single target recognition requirement, only the corresponding prompt text needs to be configured for the target recognition model, and the corresponding target recognition can be quickly achieved, thus meeting the conditions for online deployment. For example, to recognize pedestrians throwing garbage, only need to configure the prompt text "pedestrians throwing garbage", and the recognition of this type of target can be achieved. Another example, for multiple target recognition requirements, only the corresponding multiple prompt texts need to be configured for the target recognition model, and the recognition of multiple targets can be achieved, meeting the conditions for online deployment. For example, to recognize pedestrians fighting, pedestrians holding knives, and pedestrians holding children, three corresponding prompt texts can be configured, and the recognition outputs of these three types of targets can be achieved based on a single target recognition model.

[0088] Further, as Figure 5 shown, based on the foregoing embodiments, the embodiments of the present application also disclose a target recognition method. After the foregoing step 150, it further includes: step 160, step 170, and step 180.

[0089] Step 160: In response to the target recognition result indicating that the image to be recognized includes an alarm target, perform classification processing on the image to be recognized through the pre-trained classification ability of the target recognition model to obtain the target category matched by the image to be recognized.

[0090] For the image to be recognized, if it is recognized by the target recognition model that the image to be recognized includes an alarm target, then regard the image to be recognized as an alarm image. Further, based on the classification ability obtained from the first-stage training described above, perform classification processing on the alarm image to obtain the target category matched by the image to be recognized where the alarm target is recognized.

[0091] Step 170: Based on the target category and the positioning box, annotate the image to be recognized to generate incremental sample data.

[0092] The target recognition result includes: a positioning box and the confidence score of the positioning box.

[0093] Set the positioning box label matched by the image to be recognized based on the positioning box in the target recognition result and the target category, and construct an incremental sample data according to the target recognition result of the image to be recognized.

[0094] According to this method, after the target recognition model is deployed online, a large number of alarm images will be collected, and the classification ability of the target recognition model will be used for automatic annotation to obtain a number of incremental sample data.

[0095] Step 180: Fine-tune the target recognition model based on the incremental sample data.

[0096] Then, based on the incremental sample data, perform small-sample fine-tuning on the target recognition model.

[0097] In some alternative embodiments, the image feature extraction module includes: a first hybrid adapter multi-expert module and a first LoRA module, and the target extraction module includes: a second hybrid adapter multi-expert module and a second LoRA module. Fine-tuning the target recognition model based on the incremental sample data includes: freezing the model parameters in the target recognition model other than the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module, and the second LoRA module, and fine-tuning the model parameters of the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module, and the second LoRA module based on the incremental sample data. That is, freeze the remaining model parameters in the target recognition model other than the hybrid adapter multi-expert module and the LoRA module, and only train the model parameters of the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module, and the second LoRA module. And use the fine-tuned model parameters to update the model parameters of the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module, and the second LoRA module for training. And update the target recognition model with the fine-tuned model parameters.

[0098] In summary, the target recognition method disclosed in the embodiments of the present application constructs a target recognition model including a hybrid adapter multi-expert module and a LoRA module, and based on the online target recognition results, combines the text-image matching pre-training method to achieve the automatic matching of the corresponding target category and the localization box in the alarm data, realizes the online collection of incremental sample data, and performs online automatic iterative optimization on the target recognition model, occupying less training resources, and solves the problem that the training resources occupied during the training of the target recognition model increase linearly with the training data. At the same time, the model is automatically optimized online, solving the problem of high cost of manual optimization algorithms and further reducing the model maintenance cost.

[0099] Based on the foregoing embodiments, the present application also discloses a target recognition system. The system includes: a preset target recognition model 610, where the target recognition model 610 includes: an image feature extraction module, a language feature extraction module, and a target extraction module. The image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, and the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, a first output module. The system further includes:

[0100] An image acquisition unit 620, configured to acquire an image to be recognized in a target scenario;

[0101] A text acquisition unit 630, configured to acquire the prompt text in the target scenario;

[0102] The target recognition result acquisition unit 640 is configured to use the image to be recognized and the prompt text as the input of the target recognition model, and call the target recognition model 610 to perform the following target recognition steps to obtain the target recognition result corresponding to the image to be recognized: extracting image features from the image to be recognized through one or more of the first expert modules automatically adapted to the target scenario to obtain image features; and extracting language features from the prompt text through the language feature extraction module to obtain language features; performing multi-modal feature extraction on the image features and the language features through one or more of the second expert modules automatically adapted to the target scenario to obtain multi-modal features; and obtaining the target recognition result corresponding to the image to be recognized by performing feature mapping on the multi-modal features through the first output module.

[0103] Among them, the image acquisition unit 620 can sample from a video or an image to obtain an input image, and filter the input image that does not meet the quality requirements according to the target recognition requirements, for example, filtering out blurred, occluded, and pixelated lens images to obtain a to-be-recognized image that meets the requirements.

[0104] The text acquisition unit 630 can sample from the input text, filter the text that does not meet the quality requirements according to the requirements, for example, filtering out illegal, law-breaking, and meaningless text to obtain a prompt text. Then, the text is extracted as a token and used as the input of the language feature extraction module.

[0105] For the specific implementation manners of the above target recognition steps performed by calling the target recognition model 610, refer to the relevant descriptions in the foregoing embodiments, and details are not described herein again.

[0106] The target recognition system disclosed in the embodiments of the present application can perform target recognition for various target recognition requirements input by a user.

[0107] Refer to Figure 7 , the embodiments of the present application also disclose a target recognition device, and the device includes:

[0108] The target recognition model setting unit 710 is configured to preset a target recognition model, and the target recognition model includes: an image feature extraction module, a language feature extraction module, and a target extraction module. Among them, the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, and the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, and a first output module;

[0109] An input acquisition unit 720, configured to acquire an image to be recognized in a target scenario and a prompt text indicating the target to be recognized;

[0110] A first feature extraction unit 730, configured to perform image feature extraction on the image to be recognized through one or more of the first expert modules automatically adapted to the target scenario to obtain image features; and perform language feature extraction on the prompt text through the language feature extraction module to obtain language features;

[0111] A second feature extraction unit 740, configured to perform multi-modal feature extraction on the image features and the language features through one or more of the second expert modules automatically adapted to the target scenario to obtain multi-modal features;

[0112] A target recognition unit 750, configured to perform feature mapping on the multi-modal features through the first output module to obtain a target recognition result corresponding to the image to be recognized.

[0113] In some optional embodiments, the first hybrid adapter multi-expert module further includes: a first routing module, the first expert modules are connected in parallel through the first routing module, and performing image feature extraction on the image to be recognized through one or more of the first expert modules automatically adapted to the target scenario to obtain image features includes:

[0114] Invoking one or more of the first expert modules associated with the target scenario through the first routing module to obtain single-expert module output features output by the invoked first expert modules;

[0115] Fusing the single-expert module output features to obtain image features.

[0116] In some optional embodiments, the target extraction module further includes: a second routing module, the second expert modules are connected in parallel through the second routing module, and performing multi-modal feature extraction on the image features and the language features through one or more of the second expert modules automatically adapted to the target scenario to obtain multi-modal features includes:

[0117] Invoking one or more of the second expert modules associated with the target scenario through the second routing module to perform multi-modal feature extraction on the image features and the language features to obtain single-expert module multi-modal outputs;

[0118] Fusing the single-expert module multi-modal outputs to obtain multi-modal features.

[0119] In some alternative embodiments, the target recognition result includes: a bounding box. After the first output module performs feature mapping on the multi-modal features to obtain the target recognition result corresponding to the image to be recognized, the apparatus further includes:

[0120] An incremental sample data acquisition unit, configured to, in response to the target recognition result indicating that the image to be recognized includes an alarm target, perform classification processing on the image to be recognized through the pre-trained classification ability of the target recognition model, and obtain the target category matched by the image to be recognized;

[0121] The incremental sample data acquisition unit is further configured to label the image to be recognized based on the target category and the bounding box to generate incremental sample data;

[0122] A model fine-tuning unit, configured to fine-tune the target recognition model based on the incremental sample data.

[0123] In some alternative embodiments, the image feature extraction module includes: a first hybrid adapter multi-expert module and a first LoRA module, and the target extraction module includes: a second hybrid adapter multi-expert module and a second LoRA module. Fine-tuning the target recognition model based on the incremental sample data includes:

[0124] Freezing the model parameters of the target recognition model except for the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module, and the second LoRA module, and fine-tuning the model parameters of the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module, and the second LoRA module based on the incremental sample data.

[0125] In some alternative embodiments, the image feature extraction module includes a plurality of transformer structures, and the first hybrid adapter multi-expert module and the first LoRA module are located in the high-level transformer structures; the target extraction module includes a plurality of transformer structures, and the second hybrid adapter multi-expert module and the second LoRA module are located in each of the transformer structures.

[0126] In some alternative embodiments, the target recognition model is trained by the following method:

[0127] Adopt a training method of image-text matching to train the image feature extraction module and the language feature extraction module;

[0128] Freeze the model parameters of the trained image feature extraction module and the language feature extraction module, and perform a multi-modal object detection task to train the target extraction module to obtain a target recognition model.

[0129] The target recognition device disclosed in the embodiments of the present application is used to implement the target recognition method described in the embodiments of the present application. The specific implementation manners of the modules of the device will not be elaborated, and reference may be made to the specific implementation manners of the corresponding steps in the method embodiments.

[0130] The target recognition device disclosed in the embodiments of the present application includes a preset target recognition model, and the target recognition model includes: an image feature extraction module, a language feature extraction module, and a target extraction module. Among them, the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, and the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, a first output module; obtaining an image to be recognized in a target scenario and a prompt text indicating the target to be recognized; extracting image features from the image to be recognized through one or more of the first expert modules automatically adapted to the target scenario to obtain image features; and extracting language features from the prompt text through the language feature extraction module to obtain language features; extracting multi-modal features from the image features and the language features through one or more of the second expert modules automatically adapted to the target scenario to obtain multi-modal features; performing feature mapping on the multi-modal features through the first output module to obtain the target recognition result corresponding to the image to be recognized. Through the automatic selection of the large model adapter, the problem of needing to select different adapters according to different scenarios is solved. There is no need to train and deploy the target recognition model separately for the target detection requirements of each target scenario, which can greatly shorten the R & D cycle of the target recognition application and reduce the R & D and maintenance costs. By using the target recognition device disclosed in the embodiments of the present application, multiple requirements can be quickly launched based on one target recognition model, thereby solving the problems of fragmented requirements, complex algorithm maintenance, and high costs.

[0131] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference may be made to the partial description of the method embodiments for the relevant parts.

[0132] The above provides a detailed introduction to a target recognition method and device of the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

[0133] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

[0134] Each component embodiment of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the electronic device according to the embodiments of the present application. The present application can also be implemented as a device or device program (such as a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or in any other form.

[0135] For example, Figure 8 An electronic device that can implement the method according to the present application is shown. The electronic device can be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc. Traditionally, the electronic device includes a processor 810 and a memory 820, and program code 830 stored on the memory 820 and executable on the processor 810. When the processor 810 executes the program code 830, the method described in the above embodiment is implemented. The memory 820 can be a computer program product or a computer-readable medium. The memory 820 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. The memory 820 has a storage space 8201 for the program code 830 of a computer program for executing any method step in the above method. For example, the storage space 8201 for the program code 830 can include respective computer programs for implementing various steps in the above method. The program code 830 is computer-readable code. These computer programs can be read from or written into one or more computer program products. These computer program products include program code carriers such as hard disks, compact discs (CDs), memory cards, or floppy disks. The computer program includes computer-readable code, which causes the electronic device to execute the method according to the above embodiment when the computer-readable code runs on the electronic device.

[0136] The embodiments of the present application also disclose a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the target recognition method as described in the embodiments of the present application are implemented.

[0137] Such a computer program product may be a computer-readable storage medium, and the computer-readable storage medium may have storage segments, storage spaces, etc. arranged similarly to the memory 820 in the Figure 8 shown electronic device. The program code may be stored in the computer-readable storage medium in a compressed form in an appropriate manner, for example. The computer-readable storage medium is generally a portable or fixed storage unit as described in the reference Figure 9 mentioned. Generally, the storage unit includes computer-readable code 830’, and the computer-readable code 830’ is the code read by the processor, and when these codes are executed by the processor, the respective steps in the method described above are implemented.

[0138] As used herein, the terms "one embodiment", "an embodiment", or "one or more embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. In addition, it should be noted that the examples of the phrase "in one embodiment" herein do not necessarily all refer to the same embodiment.

[0139] In the specification provided herein, a large number of specific details are set forth. However, it can be understood that the embodiments of the present application may be practiced without these specific details. In some instances, well-known methods, structures, and technologies have not been shown in detail so as not to obscure the understanding of this specification.

[0140] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim listing several devices, several of these devices may be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not denote any order. These words may be interpreted as names.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. A target recognition method, characterized in that, The method includes: Presetting a target recognition model, where the target recognition model includes an image feature extraction module, a language feature extraction module, and a target extraction module. Among them, the image feature extraction module includes a first hybrid adapter multi-expert module composed of multiple different first expert modules, and the target extraction module includes a second hybrid adapter multi-expert module composed of multiple different second expert modules and a first output module; Obtaining an image to be recognized in a target scenario and a prompt text indicating the target to be recognized; Performing image feature extraction on the image to be recognized through one or more of the first expert modules automatically adapted to the target scenario to obtain image features; and performing language feature extraction on the prompt text through the language feature extraction module to obtain language features; Performing multi-modal feature extraction on the image features and the language features through one or more of the second expert modules automatically adapted to the target scenario to obtain multi-modal features; Performing feature mapping on the multi-modal features through the first output module to obtain a target recognition result corresponding to the image to be recognized; The target recognition result includes a bounding box. After performing feature mapping on the multi-modal features through the first output module to obtain the target recognition result corresponding to the image to be recognized, it further includes: In response to the target recognition result indicating that the image to be recognized includes an alarm target, performing classification processing on the image to be recognized through the pre-trained classification ability of the target recognition model to obtain a target category matching the image to be recognized; Annotating the image to be recognized based on the target category and the bounding box to generate incremental sample data; Fine-tuning the target recognition model based on the incremental sample data.

2. The method according to claim 1, wherein The first hybrid adapter multi-expert module further includes a first routing module. The first expert modules are connected in parallel through the first routing module. Performing image feature extraction on the image to be recognized through one or more of the first expert modules automatically adapted to the target scenario to obtain image features includes: Invoking one or more of the first expert modules associated with the target scenario through the first routing module to obtain single-expert module output features output by the invoked first expert modules; Fusing the single-expert module output features to obtain image features.

3. The method according to claim 1, wherein The target extraction module further includes a second routing module. The second expert modules are connected in parallel through the second routing module. Performing multi-modal feature extraction on the image features and the language features through one or more of the second expert modules automatically adapted to the target scenario to obtain multi-modal features includes: Invoking one or more of the second expert modules associated with the target scenario through the second routing module to perform multi-modal feature extraction on the image features and the language features to obtain single-expert module multi-modal outputs; Fusing the single-expert module multi-modal outputs to obtain multi-modal features.

4. The method according to claim 1, wherein The image feature extraction module includes: a first hybrid adapter multi-expert module and a first LoRA module, the target extraction module includes: a second hybrid adapter multi-expert module and a second LoRA module, and fine-tuning the target recognition model based on the incremental sample data includes: Freeze the model parameters of the target recognition model except for the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module, and the second LoRA module, and fine-tune the model parameters of the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module, and the second LoRA module based on the incremental sample data.

5. The method according to claim 4, wherein The image feature extraction module includes multiple transformer structures, and the first hybrid adapter multi-expert module and the first LoRA module are located in the high-level transformer structures; the target extraction module includes multiple transformer structures, and the second hybrid adapter multi-expert module and the second LoRA module are located in each of the transformer structures.

6. The method according to claim 1, wherein The target recognition model is trained by the following method: Adopt a training method of image-text matching to train the image feature extraction module and the language feature extraction module; Freeze the model parameters of the trained image feature extraction module and the language feature extraction module, and perform a multi-modal object detection task to train the target extraction module to obtain a target recognition model.

7. A target recognition device, characterized in that, The device includes: A target recognition model setting unit for presetting a target recognition model, the target recognition model includes: an image feature extraction module, a language feature extraction module, and a target extraction module, wherein the image feature extraction module includes: a first hybrid adapter multi-expert module composed of multiple different first expert modules, and the target extraction module includes: a second hybrid adapter multi-expert module composed of multiple different second expert modules, and a first output module; An input acquisition unit for acquiring an image to be recognized in a target scenario and a prompt text indicating the target to be recognized; A first feature extraction unit for extracting image features from the image to be recognized through one or more of the first expert modules automatically adapted to the target scenario to obtain image features; and extracting language features from the prompt text through the language feature extraction module to obtain language features; A second feature extraction unit for performing multi-modal feature extraction on the image features and the language features through one or more of the second expert modules automatically adapted to the target scenario to obtain multi-modal features; A target recognition unit for performing feature mapping on the multi-modal features through the first output module to obtain a target recognition result corresponding to the image to be recognized; The target recognition result includes: a bounding box. After obtaining the target recognition result corresponding to the image to be recognized by performing feature mapping on the multi-modal features through the first output module, the device further includes: An incremental sample data acquisition unit, configured to, in response to the target recognition result indicating that the image to be recognized includes an alarm target, perform classification processing on the image to be recognized through the pre-trained classification ability of the target recognition model, and obtain the target category matched by the image to be recognized; The incremental sample data acquisition unit is further configured to label the image to be recognized based on the target category and the positioning frame to generate incremental sample data; A model fine-tuning unit, configured to fine-tune the target recognition model based on the incremental sample data.

8. A target recognition system, characterized in that, The system includes: a preset target recognition model, where the target recognition model includes: an image feature extraction module, a language feature extraction module, and a target extraction module. The image feature extraction module includes: a first hybrid adapter multi-expert module composed of multiple different first expert modules. The target extraction module includes: a second hybrid adapter multi-expert module composed of multiple different second expert modules, a first output module. The system further includes: An image acquisition unit, configured to acquire an image to be recognized in a target scenario; A text acquisition unit, configured to acquire a prompt text in the target scenario; A target recognition result acquisition unit, configured to use the image to be recognized and the prompt text as inputs of the target recognition model, call the target recognition model to perform the following target recognition steps, and obtain the target recognition result corresponding to the image to be recognized: extract image features from the image to be recognized through one or more of the first expert modules automatically adapted to the target scenario to obtain image features; and extract language features from the prompt text through the language feature extraction module to obtain language features; perform multi-modal feature extraction on the image features and the language features through one or more of the second expert modules automatically adapted to the target scenario to obtain multi-modal features; perform feature mapping on the multi-modal features through the first output module to obtain the target recognition result corresponding to the image to be recognized; The target recognition result includes: a positioning frame. After performing feature mapping on the multi-modal features through the first output module to obtain the target recognition result corresponding to the image to be recognized, the following is further included: In response to the target recognition result indicating that the image to be recognized includes an alarm target, perform classification processing on the image to be recognized through the pre-trained classification ability of the target recognition model, and obtain the target category matched by the image to be recognized; Label the image to be recognized based on the target category and the positioning frame to generate incremental sample data; Fine-tune the target recognition model based on the incremental sample data.

9. An electronic device, comprising a memory, a processor, and program code stored on the memory and executable on the processor, characterized in that, The processor implements the method according to any one of claims 1 to 6 when executing the program code.

10. A computer-readable storage medium having program code stored thereon, characterized in that, The steps of the method according to any one of claims 1 to 6 are implemented when the program code is executed by the processor.

Images