Target identification method, device and system, electronic equipment and storage medium

Abstract

The invention discloses a target recognition method, device and system and electronic equipment, and belongs to the technical field of artificial intelligence. The method comprises the steps that a target recognition model comprising an image feature extraction module, a language feature extraction module and a target extraction module is preset, and the image feature extraction module comprises a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, the target extraction module comprises a second hybrid adapter multi-expert module composed of a plurality of second expert modules; obtaining a to-be-recognized image and a prompt text in the target scene; an automatically-adaptive first expert module extracts image features of the to-be-recognized image; extracting language features of the prompt text through a language feature extraction module; and a second expert module is automatically adapted to extract multi-modal features of the image features and the language features, and a target recognition result corresponding to the to-be-recognized image is obtained based on the multi-modal features. According to the method, multiple target identification can be realized through one model, and the research and development period of related applications is shortened.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a target recognition method, a target recognition device, a target recognition system, an electronic device, and a computer-readable storage medium. Background Art

[0002] "Long-tail targets" refer to targets with a low probability of occurrence, including but not limited to: events, behaviors, people, animals, etc., such as fierce dogs, riding against the flow of traffic, violent incidents, etc. With the improvement of security camera hardware performance, resolution, and the development of artificial intelligence technology, security cameras and artificial intelligence algorithms can identify a variety of targets in cities and villages to ensure public security. For example, security cameras can be used to collect surveillance images, and artificial intelligence technology can be used to identify whether there are targets that violate public security or urban environmental requirements in the collected images, which can be used for subsequent alarms, public security management and other applications.

[0003] However, the existing technology has various requirements for the recognition of long-tail targets, and there is a trend of customization and fragmentation. For example, some requirements are to identify women holding children, some requirements are to identify middle-aged men wearing black pants holding children, and some requirements are to identify large dogs. For various recognition requirements, the long-tail targets are different. When the recognition algorithm based on traditional artificial intelligence methods recognizes the long-tail targets of a specified scene, it is necessary to customize the development for the recognition requirements of these long-tail targets, which has the defects of difficult data collection, long R&D cycle, high R&D cost, and high cost of manual optimization algorithm.

[0004] It can be seen that the target recognition method in the prior art still needs to be improved. Summary of the invention

[0005] The embodiments of the present application provide a target recognition method and device, which help to shorten the research and development cycle of target recognition applications and reduce research and development and maintenance costs.

[0006] In a first aspect, an embodiment of the present application provides a target recognition method, comprising:

[0007] A preset target recognition model, the target recognition model includes: an image feature extraction module, a language feature extraction module and a target extraction module, wherein the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, and a first output module;

[0008] Obtaining an image to be identified in a target scene and a prompt text indicating a target to be identified;

[0009] Performing image feature extraction on the image to be recognized to obtain image features through one or more of the first expert modules that are automatically adapted to the target scene; and performing language feature extraction on the prompt text to obtain language features through the language feature extraction module;

[0010] Performing multimodal feature extraction on the image features and the language features through one or more second expert modules that are automatically adapted to the target scene to obtain multimodal features;

[0011] The first output module performs feature mapping on the multimodal features to obtain a target recognition result corresponding to the image to be recognized.

[0012] In a second aspect, an embodiment of the present application provides a target recognition device, including:

[0013] The target recognition model setting unit is used to preset the target recognition model, wherein the target recognition model includes: an image feature extraction module, a language feature extraction module and a target extraction module, wherein the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, and the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, and a first output module;

[0014] An input acquisition unit, used to acquire an image to be recognized in a target scene and a prompt text indicating a target to be recognized;

[0015] A first feature extraction unit is used to extract image features from the image to be identified by using one or more of the first expert modules that are automatically adapted to the target scene to obtain image features; and to extract language features from the prompt text by using the language feature extraction module to obtain language features;

[0016] a second feature extraction unit, configured to perform multimodal feature extraction on the image feature and the language feature by using one or more second expert modules automatically adapted to the target scene to obtain a multimodal feature;

[0017] The target recognition unit is used to perform feature mapping on the multimodal features through the first output module to obtain a target recognition result corresponding to the image to be recognized.

[0018] In a third aspect, an embodiment of the present application provides a target recognition system, the system comprising: a preset target recognition model, wherein the target recognition model comprises: an image feature extraction module, a language feature extraction module and a target extraction module, wherein the image feature extraction module comprises: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, the target extraction module comprises: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, a first output module, and the system further comprises:

[0019] An image acquisition unit, used for acquiring an image to be recognized in a target scene;

[0020] A text collection unit, used to collect prompt text in the target scene;

[0021] The target recognition result acquisition unit is used to use the image to be recognized and the prompt text as inputs of the target recognition model, call the target recognition model to execute the following target recognition steps, and obtain the target recognition result corresponding to the image to be recognized: extract image features of the image to be recognized through one or more of the first expert modules that are automatically adapted to the target scene to obtain image features; and extract language features of the prompt text through the language feature extraction module to obtain language features; extract multimodal features of the image features and the language features through one or more of the second expert modules that are automatically adapted to the target scene to obtain multimodal features; and perform feature mapping on the multimodal features through the first output module to obtain the target recognition result corresponding to the image to be recognized.

[0022] In a fourth aspect, an embodiment of the present application further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the target recognition method described in the embodiment of the present application when executing the computer program.

[0023] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the target identification method disclosed in the embodiment of the present application are implemented.

[0024] The target recognition method disclosed in the embodiment of the present application presets a target recognition model, wherein the target recognition model includes: an image feature extraction module, a language feature extraction module and a target extraction module, wherein the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, and the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, and a first output module; obtains an image to be recognized in a target scene and a prompt text indicating a target to be recognized; extracts image features of the image to be recognized through one or more of the first expert modules automatically adapted to the target scene to obtain image features; and The language feature extraction module extracts language features from the prompt text to obtain language features; through one or more of the second expert modules that are automatically adapted to the target scene, multimodal feature extraction is performed on the image features and the language features to obtain multimodal features; through the first output module, feature mapping is performed on the multimodal features to obtain the target recognition result corresponding to the image to be recognized. Through the automatic selection of the large model adapter, the problem of needing to select different adapters according to different scenes is solved. There is no need to train and deploy target recognition models for the target detection requirements of each target scene, which can greatly shorten the research and development cycle of target recognition applications and reduce research and development and maintenance costs. The target recognition method disclosed in the embodiment of the present application is adopted to realize the rapid launch of multiple requirements based on one target recognition model, thereby solving the problems of fragmented requirements, complex algorithm maintenance, and high cost.

[0025] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] Figure 1 It is one of the flow charts of the target recognition method disclosed in the embodiment of the present application;

[0028] Figure 2 It is one of the structural schematic diagrams of the target recognition model disclosed in the embodiment of the present application;

[0029] Figure 3 This is the second structural diagram of the target recognition model disclosed in the embodiment of the present application;

[0030] Figure 4 It is a schematic diagram of the structure of the first hybrid adapter multi-expert module disclosed in the embodiment of the present application;

[0031] Figure 5 This is the second flow chart of the target recognition method disclosed in the embodiment of the present application;

[0032] Figure 6 is a schematic diagram of the structure of the target recognition system disclosed in the embodiment of the present application;

[0033] Figure 7 is a schematic diagram of the structure of the target recognition device disclosed in the embodiment of the present application;

[0034] Figure 8 A block diagram schematically shows an electronic device for executing the method according to the present application; and

[0035] Fig. 9 A storage unit for holding or carrying a program code for implementing the method according to the present application is schematically shown. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0037] Reference Figure 1 , a target recognition method disclosed in an embodiment of the present application includes: steps 110 to 150.

[0038] Step 110, preset a target recognition model, the target recognition model includes: an image feature extraction module, a language feature extraction module and a target extraction module, wherein the image feature extraction module includes: a first hybrid adapter multi-expert module composed of multiple different first expert modules, the target extraction module includes: a second hybrid adapter multi-expert module composed of multiple different second expert modules, and a first output module.

[0039] like Figure 2 As shown, the target recognition model at least includes: an image feature extraction module 210, a language feature extraction module 220 and a target extraction module 230, wherein the outputs of the image feature extraction module 210 and the language feature extraction module 220 are used together as the input of the target extraction module 230.

[0040] The image feature extraction module 210 is used to extract image features of an input image, the language feature extraction module 220 is used to extract language features of an input text, and the target extraction module 230 is used to perform target recognition based on the input image features and language features.

[0041] In some optional embodiments, such as Figure 3 As shown, the target recognition model also includes: a classification module 240, the outputs of the image feature extraction module 210 and the language feature extraction module 220 are used as inputs of the classification module 240, and the classification module 240 is used to classify targets based on the input image features and language features.

[0042] The image feature extraction module 210 is constructed based on the VIT (Vision Transformer) model. During operation, the image feature extraction module 210 divides the input image into a sequence of image blocks, and performs feature extraction based on the sequence of image blocks, and outputs low-level visual features (texture, color, etc.) and high-level semantic features.

[0043] In some optional embodiments, the image feature extraction module 210 includes multiple transformers (transformer structures), and the first hybrid adapter multi-expert module and the first LoRA module are located in a high-level transformer structure. The first hybrid adapter multi-expert module and the first LoRA module are arranged in a high-level transformer structure, which can support fine-tuning of the target recognition model and incremental learning while avoiding the occurrence of catastrophic forgetting.

[0044] In some optional embodiments, the first hybrid adapter multi-expert module includes a plurality of different first expert modules. The first hybrid adapter multi-expert module also includes: a first routing module, and the first expert modules are connected in parallel through the first routing module. Figure 4 In the structure of the first hybrid adapter multi-expert module shown, the first expert module can be implemented as FFN, the first routing module can adopt a softmax structure, and the first expert module can be connected in parallel through the first routing module in a residual manner. The first routing module is used to control the call to the first expert module.

[0045] by Figure 4Taking the structure of the first hybrid adapter multi-expert module shown as an example, the first expert module corresponding to each FFN (Feed-Forward Neural Network) can be used to perform feature processing of different functions, and the first routing module of the softmax structure is used to specifically call several FFNs according to the task control corresponding to the current scene, and fuse the features of the called FFN outputs as the output of the transformer.

[0046] The role of FFN is to perform a fully connected transformation on the position encoded or decoded vector to strengthen the representation of position information and feature extraction. The first hybrid adapter multi-expert module 210 can better capture the global dependency of the input sequence by increasing the depth and width of the feedforward neural network, thereby improving the accuracy of feature extraction.

[0047] In some optional embodiments, the first LoRA module is located on the Q and V matrices of the transformer's self-attention mechanism. When training the image feature extraction module 210, the first LoRA module is connected to Q and V by residual means, and is added to the original Q and V outputs as the outputs of Q and V. During inference, the first LoRA module is directly added to the original Q and V matrices as the outputs of Q and V.

[0048] In some optional embodiments, the language feature extraction module can be constructed based on the BERT model. The language feature extraction module is used to extract the language features of the input text. Among them, the text input to the language feature extraction module can be a description text of the recognition target. The specific implementation of the language feature extraction module refers to the prior art and will not be repeated in the embodiments of this application.

[0049] Optionally, the target extraction module 230 includes: a second hybrid adapter multi-expert module located between the first output module and the ... first output module. The target extraction module 230 includes a plurality of transformer structures, and the second hybrid adapter multi-expert module is located in each of the transformer structures.

[0050] In some optional embodiments, the second hybrid adapter multi-expert module includes a plurality of different second expert modules. The second hybrid adapter multi-expert module also includes: a second routing module, and the second expert modules are connected in parallel through the second routing module. The structure of the second hybrid adapter multi-expert module refers to the structure of the first hybrid adapter multi-expert module, which will not be repeated here.

[0051] The second expert module can be implemented as FFN, the second routing module can adopt a softmax structure, and the second expert module can be connected in parallel through the second routing module in a residual manner. The second routing module is used to control the call of the second expert module. The second expert module corresponding to each FFN structure can be used to perform feature processing of different functions. The second routing module of the softmax structure is used to specifically call several FFNs according to the task control corresponding to the current scene, and fuse the features of the called FFN output to obtain multimodal features as the output of the transformer.

[0052] Optionally, the first output module is constructed based on a fully connected layer. The multimodal features obtained after fusion are input to the first output module for feature mapping to obtain a target recognition result. For example, the first output module includes two fully connected layers, and the multimodal features are used as inputs of the two fully connected layers respectively, and the two fully connected layers respectively output the positioning box of the target and the confidence score of the positioning result.

[0053] In some optional embodiments, the target extraction module 230 further includes: a second LoRA module, the second LoRA module is located on the Q and V matrices of the self-attention mechanism of each transformer. The combination of the second LoRA module and the Q and V matrices is described above and will not be repeated here.

[0054] The target extraction module 230 stacks multiple transformers and fuses image features and language features at different levels through an attention mechanism, and finally outputs corresponding features.

[0055] In some optional embodiments, the target recognition model is trained by the following method: performing classification tasks by using a picture-text matching training method to train the image feature extraction module and the language feature extraction module; freezing the model parameters of the image feature extraction module and the language feature extraction module obtained by training, performing a multimodal target detection task to train the target extraction module, and obtaining a target recognition model.

[0056] In an embodiment of the present application, a two-stage training method is adopted to train the target recognition model.

[0057] In the first stage, the image feature extraction module and the language feature extraction module are trained by training the classification capability model and adopting a picture-text matching training method.

[0058] Optionally, a picture-text matching training method is used to train the image feature extraction module and the language feature extraction module, including: using a picture-text matching training method to perform an image classification task on the first training data set, and training the image feature extraction module and the language feature extraction module.

[0059] First, we collect a wide range of graphic and text data from the Internet and public data sets, with a data volume of billions. These data contain most of the goals and behaviors that have appeared in human society. For example, we set category description text for the pictures and construct the first training data set based on the acquired data.

[0060] Afterwards, the image feature extraction module and the language feature extraction module are used to extract features of the image and the description text in the first training data set respectively, and the cosine similarity of the features output by the image feature extraction module and the features output by the language feature extraction module is calculated. The image feature extraction module and the language feature extraction module are trained by training the image-text matching capability.

[0061] The image-text matching capability of the target recognition model expresses the similarity between the image and the category description text, that is, if the image A matches the description text T, it can be considered that the target in the image A belongs to the target category corresponding to the description text T. That is, the process of training the image feature extraction module and the language feature extraction module simultaneously trains the image classification capability of the target recognition model.

[0062] The specific implementation method of training the image feature extraction module and the language feature extraction module by using the image-text matching training method can refer to the training method of the clip model in the prior art, which will not be repeated here.

[0063] In the second stage, the target extraction module is trained.

[0064] Optionally, a second training dataset is constructed by widely collecting public target recognition datasets, such as coco and grounding datasets (training datasets for computer vision and natural language models), converting them into a data form of description text + positioning box.

[0065] Afterwards, the model parameters of the image feature extraction module and the language feature extraction module that have completed training are frozen, and the model parameters of the target extraction module are trained. The training process is as follows: the image feature extraction module and the language feature extraction module are used to extract image features and language features of the training samples in the second training data set respectively; then, the extracted image features and language features are input into the target extraction module to perform multimodal feature extraction and feature fusion mapping to obtain the category and positioning box of the target in the training sample; then, based on the loss between the category description text and the target positioning box annotated on the training sample and the category and positioning box output by the target extraction module, the model parameters of the target extraction module are fine-tuned, thereby iteratively training the target extraction module.

[0066] By combining the two-stage training method of image-text matching training and multimodal target extraction, the target recognition model has both image classification and target detection and recognition capabilities.

[0067] For the pre-trained object recognition model, when it is deployed for inference, when an image and a prompt text describing the object to be recognized are input, the object recognition model can infer the positioning box of the object in the image.

[0068] Step 120: Acquire the image to be identified in the target scene and the prompt text indicating the target to be identified.

[0069] The target scenarios include, but are not limited to, specified behavior recognition (such as driving against traffic, littering, etc.), large dog recognition, etc.

[0070] The image to be identified in the target scene can be obtained by using image acquisition technology. The specific implementation method of obtaining the image to be identified in the target scene is not limited in the embodiment of the present application. The prompt text is set according to the target recognition requirements. Taking the scene of identifying wrong-way behavior as an example, the prompt text can be "wrong-way vehicle".

[0071] Step 130, extracting image features of the image to be identified by one or more of the first expert modules that are automatically adapted to the target scene to obtain image features; and extracting language features of the prompt text by the language feature extraction module to obtain language features.

[0072] As described above, the image feature extraction module includes: a first hybrid adapter multi-expert module, wherein the first hybrid adapter multi-expert module further includes: a plurality of the first expert modules and a first routing module. Accordingly, in some optional embodiments, the image feature extraction of the image to be identified and the acquisition of image features through one or more of the first expert modules that are automatically adapted to the target scene include: calling one or more of the first expert modules associated with the target scene through the first routing module to acquire the single expert module output features output by the called first expert modules; and fusing the single expert module output features to acquire image features.

[0073] Optionally, one or more of the first expert modules associated with the target scenario may be preset according to the test result of the target recognition model. For example, the first expert modules respectively called under different target scenarios may be preset.

[0074] In some other optional embodiments, one or more of the first expert modules associated with the target scene can be determined based on the calling weights of the first expert modules obtained by training the target recognition model. For example, when training target recognition models based on training data of different target scenes, the weights of the first expert modules in the target recognition model for each target scene are recorded, and in the target recognition stage, the first routing module calls the corresponding first expert module according to the weights determined in the training stage.

[0075] In a specific implementation, after the image to be identified is input into the image feature extraction module, the first routing module in the image feature extraction module automatically selects one or more of the first expert modules to perform feature extraction on the image to be identified respectively, and outputs corresponding image features. Afterwards, the image features output by the selected first expert modules are fused as the image features output by the image feature extraction module.

[0076] On the other hand, the prompt text is input into the language feature extraction module, and the language feature extraction module extracts the language feature of the prompt text and outputs the language feature of the prompt text.

[0077] Step 140 , performing multimodal feature extraction on the image features and the language features by using one or more of the second expert modules that are automatically adapted to the target scene to obtain multimodal features.

[0078] Next, the image features output by the image feature extraction module and the language features output by the language feature extraction module are input to the object extraction module.

[0079] As described above, the target extraction module includes: the second hybrid adapter multi-expert module and the first output module, wherein the second hybrid adapter multi-expert module includes: a plurality of different second expert modules and a second routing module, and the second expert modules are connected in parallel through the second routing module. Accordingly, the multimodal feature extraction of the image features and the language features through one or more second expert modules that are automatically adapted to the target scene to obtain multimodal features includes: calling one or more second expert modules associated with the target scene through the second routing module to extract multimodal features of the image features and the language features to obtain a single expert module multimodal output; fusing the single expert module multimodal output to obtain multimodal features.

[0080] Among them, the method for determining one or more second expert modules associated with the target scene can be found in the method for determining one or more first expert modules associated with the target scene described above, which will not be repeated here.

[0081] In a specific implementation, after the image features and the language features are input into the target extraction module, the second routing module in the target extraction module automatically selects one or more of the second expert modules to perform feature processing on the image features and the language features respectively, and outputs the corresponding feature processing results as the multimodal output of the single expert module. Afterwards, the second routing module fuses the multimodal outputs of each of the single expert modules to obtain fused features as the multimodal features extracted from the image features and the language features.

[0082] Step 150: Perform feature mapping on the multimodal features through the first output module to obtain a target recognition result corresponding to the image to be recognized.

[0083] Optionally, the first output module includes two fully connected layers, wherein one fully connected layer is used to perform feature mapping processing on the multimodal features to obtain the positioning frame of the target to be identified; and the other fully connected layer is used to perform feature mapping processing on the multimodal features to obtain the positioning result confidence score of the corresponding positioning frame. Further, according to the positioning result confidence score, the positioning frame of the target to be identified can be determined. For example, the positioning frame with the highest confidence score is selected as the positioning frame of the target to be identified.

[0084] The positioning box and the confidence level are used as target recognition results.

[0085] Furthermore, according to the target to be identified indicated by the prompt text, it can be determined whether the current image to be identified is a warning image. That is, if a warning target is identified in the image to be identified, the image to be identified is a warning image.

[0086] In summary, the target recognition method disclosed in the embodiments of the present application presets a target recognition model, wherein the target recognition model includes: an image feature extraction module, a language feature extraction module and a target extraction module, wherein the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, and the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, and a first output module; obtains an image to be recognized in a target scene and a prompt text indicating a target to be recognized; extracts image features of the image to be recognized through one or more of the first expert modules that are automatically adapted to the target scene to obtain image features; and, The language feature extraction module extracts the language features of the prompt text to obtain the language features; the one or more second expert modules that are automatically adapted to the target scene extract the multimodal features of the image features and the language features to obtain the multimodal features; the first output module performs feature mapping on the multimodal features to obtain the target recognition result corresponding to the image to be recognized. The automatic selection of the large model adapter solves the problem of needing to select different adapters according to different scenes. It is not necessary to train and deploy target recognition models for the target detection requirements of each target scene, which can greatly shorten the research and development cycle of target recognition applications and reduce research and development and maintenance costs. The target recognition method disclosed in the embodiment of the present application is used to quickly launch multiple requirements based on one target recognition model, thereby solving the problems of fragmented requirements, complex algorithm maintenance, and high costs.

[0087] For example, after training the target recognition model, for a specific single target recognition requirement, you only need to configure the corresponding prompt text for the target recognition model to quickly realize the corresponding target recognition, so that the conditions for online deployment are met. For example, to identify pedestrians who throw garbage, you only need to configure a prompt text such as "pedestrians who throw garbage" to realize the recognition of this type of target. For another example, for multiple target recognition requirements, you only need to configure the target recognition model with corresponding multiple prompt texts to realize the recognition of multiple targets and meet the conditions for online deployment. For example, to identify pedestrians fighting, pedestrians holding knives, and pedestrians holding children, you can configure three corresponding prompt texts, and the recognition output of these three targets can be realized based on one target recognition model.

[0088] Further, such as Figure 5 As shown, based on the above-mentioned embodiment, the embodiment of the present application further discloses a target recognition method, which, after the above-mentioned step 150, also includes: step 160, step 170 and step 180.

[0089] Step 160, in response to the target recognition result indicating that the image to be recognized includes an alarm target, the image to be recognized is classified by using the pre-trained classification capability of the target recognition model to obtain a target category matched by the image to be recognized.

[0090] For the image to be identified, if the target recognition model recognizes that the image to be identified includes an alarm target, the image to be identified is used as an alarm image. Furthermore, based on the classification ability obtained in the first stage of training mentioned above, the alarm image is classified to obtain the target category that matches the image to be identified that identifies the alarm target.

[0091] Step 170: annotate the image to be identified based on the target category and the positioning box to generate incremental sample data.

[0092] The target recognition result includes: a positioning box and a confidence score of the positioning box.

[0093] A positioning frame label matching the image to be identified is set based on the positioning frame in the target recognition result and the target category, and an incremental sample data is constructed according to the target recognition result of the image to be identified.

[0094] According to this method, after the target recognition model is deployed online, a large number of warning images will be collected and automatically labeled using the classification capability of the target recognition model to obtain a number of incremental sample data.

[0095] Step 180: fine-tune the target recognition model based on the incremental sample data.

[0096] Then, based on the incremental sample data, the target recognition model is fine-tuned with small samples.

[0097] In some optional embodiments, the image feature extraction module includes: a first hybrid adapter multi-expert module and a first LoRA module, the target extraction module includes: a second hybrid adapter multi-expert module and a second LoRA module, and the fine-tuning of the target recognition model based on the incremental sample data includes: freezing the model parameters of the target recognition model except the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module and the second LoRA module, and fine-tuning the model parameters of the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module and the second LoRA module based on the incremental sample data. That is, the remaining model parameters except the hybrid adapter multi-expert module and the LoRA module in the target recognition model are frozen, and only the model parameters of the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module and the second LoRA module are trained. The fine-tuned model parameters are used to update the model parameters of the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module and the second LoRA module for training. The fine-tuned model parameters are used to update the target recognition model.

[0098] In summary, the target recognition method disclosed in the embodiment of the present application constructs a target recognition model including a hybrid adapter multi-expert module and a LoRA module, and based on the online target recognition results, combines the image-text matching pre-training method to achieve automatic matching of the corresponding target category in the alarm data with the positioning box, realizes online collection of incremental sample data, and performs online automatic iterative optimization of the target recognition model, which occupies less training resources and solves the problem that the training resources occupied during the training of the target recognition model grow linearly with the training data. At the same time, the model is automatically optimized online, which solves the problem of high cost of manual optimization algorithms and further reduces the cost of model maintenance.

[0099] Based on the above embodiments, the present application also discloses a target recognition system, the system comprising: a preset target recognition model 610, wherein the target recognition model 610 comprises: an image feature extraction module, a language feature extraction module and a target extraction module, wherein the image feature extraction module comprises: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, the target extraction module comprises: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, a first output module, and the system further comprises:

[0100] An image acquisition unit 620 is used to acquire an image to be recognized in a target scene;

[0101] A text collection unit 630, used to collect prompt text in the target scene;

[0102] The target recognition result acquisition unit 640 is used to use the image to be recognized and the prompt text as inputs of the target recognition model, call the target recognition model 610 to execute the following target recognition steps, and obtain the target recognition result corresponding to the image to be recognized: extract image features of the image to be recognized through one or more of the first expert modules that are automatically adapted to the target scene to obtain image features; and extract language features of the prompt text through the language feature extraction module to obtain language features; extract multimodal features of the image features and the language features through one or more of the second expert modules that are automatically adapted to the target scene to obtain multimodal features; and perform feature mapping on the multimodal features through the first output module to obtain the target recognition result corresponding to the image to be recognized.

[0103] Among them, the image acquisition unit 620 can sample from the video or image to obtain the input image, and filter the input image that does not meet the quality requirements according to the target recognition requirements, for example, filtering out blurred, blocked, and distorted lens images to obtain an image to be recognized that meets the requirements.

[0104] The text collection unit 630 can sample from the input text and filter the text that does not meet the quality requirements according to the requirements, such as filtering out the text that is illegal, illegal, and has no semantic meaning, to obtain the prompt text. After that, the text is extracted into tokens as the input of the language feature extraction module.

[0105] For the specific implementation of the above target recognition step performed by calling the target recognition model 610, please refer to the relevant description in the previous embodiment, which will not be repeated here.

[0106] The target recognition system disclosed in the embodiment of the present application can perform target recognition according to various target recognition requirements input by the user.

[0107] Reference Figure 7 The present application also discloses a target recognition device, which includes:

[0108] The target recognition model setting unit 710 is used to preset the target recognition model, wherein the target recognition model includes: an image feature extraction module, a language feature extraction module and a target extraction module, wherein the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, and the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, and a first output module;

[0109] An input acquisition unit 720 is used to acquire an image to be recognized in a target scene and a prompt text indicating a target to be recognized;

[0110] The first feature extraction unit 730 is used to extract image features from the image to be recognized by using one or more of the first expert modules that are automatically adapted to the target scene to obtain image features; and to extract language features from the prompt text by using the language feature extraction module to obtain language features;

[0111] A second feature extraction unit 740, configured to perform multimodal feature extraction on the image feature and the language feature by using one or more second expert modules that are automatically adapted to the target scene to obtain a multimodal feature;

[0112] The target recognition unit 750 is used to perform feature mapping on the multimodal features through the first output module to obtain a target recognition result corresponding to the image to be recognized.

[0113] In some optional embodiments, the first hybrid adapter multi-expert module further includes: a first routing module, the first expert modules are connected in parallel through the first routing module, and the one or more first expert modules automatically adapted to the target scene are used to extract image features from the image to be identified to obtain image features, including:

[0114] Calling one or more of the first expert modules associated with the target scene through the first routing module to obtain single expert module output features output by the called first expert modules;

[0115] The output features of the single expert module are integrated to obtain image features.

[0116] In some optional embodiments, the target extraction module further includes: a second routing module, the second expert modules are connected in parallel through the second routing module, and the multimodal feature extraction is performed on the image features and the language features through one or more second expert modules that are automatically adapted to the target scene to obtain multimodal features, including:

[0117] Calling one or more of the second expert modules associated with the target scene through the second routing module, performing multimodal feature extraction on the image features and the language features, and obtaining a single expert module multimodal output;

[0118] The multimodal outputs of the single expert module are fused to obtain multimodal features.

[0119] In some optional embodiments, the target recognition result includes: a positioning frame, and after the first output module performs feature mapping on the multimodal feature to obtain the target recognition result corresponding to the image to be recognized, the device further includes:

[0120] an incremental sample data acquisition unit, configured to classify the image to be identified by using the pre-trained classification capability of the target recognition model in response to the target recognition result indicating that the image to be identified includes a warning target, and obtain a target category matched by the image to be identified;

[0121] The incremental sample data acquisition unit is further used to mark the image to be identified based on the target category and the positioning frame to generate incremental sample data;

[0122] A model fine-tuning unit is used to fine-tune the target recognition model based on the incremental sample data.

[0123] In some optional embodiments, the image feature extraction module includes: a first hybrid adapter multi-expert module and a first LoRA module, the target extraction module includes: a second hybrid adapter multi-expert module and a second LoRA module, and the target recognition model is fine-tuned based on the incremental sample data, including:

[0124] Freeze the model parameters of the target recognition model except the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module and the second LoRA module, and fine-tune the model parameters of the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module and the second LoRA module based on the incremental sample data.

[0125] In some optional embodiments, the image feature extraction module includes multiple transformer structures, the first hybrid adapter multi-expert module and the first LoRA module are located in a high-level transformer structure; the target extraction module includes multiple transformer structures, the second hybrid adapter multi-expert module and the second LoRA module are located in each of the transformer structures.

[0126] In some optional embodiments, the target recognition model is trained by the following method:

[0127] Using a picture-text matching training method to train the image feature extraction module and the language feature extraction module;

[0128] The model parameters of the image feature extraction module and the language feature extraction module obtained through training are frozen, and a multimodal target detection task is performed to train the target extraction module to obtain a target recognition model.

[0129] The target recognition device disclosed in the embodiment of the present application is used to implement the target recognition method described in the embodiment of the present application. The specific implementation methods of each module of the device will not be repeated here. Please refer to the specific implementation methods of the corresponding steps in the method embodiment.

[0130] The target recognition device disclosed in the embodiment of the present application presets a target recognition model, wherein the target recognition model includes: an image feature extraction module, a language feature extraction module and a target extraction module, wherein the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, and the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, and a first output module; obtains an image to be recognized in a target scene and a prompt text indicating a target to be recognized; extracts image features of the image to be recognized through one or more of the first expert modules automatically adapted to the target scene to obtain image features; and The language feature extraction module extracts language features from the prompt text to obtain language features; through one or more of the second expert modules that are automatically adapted to the target scene, multimodal feature extraction is performed on the image features and the language features to obtain multimodal features; through the first output module, feature mapping is performed on the multimodal features to obtain the target recognition result corresponding to the image to be recognized. Through the automatic selection of the large model adapter, the problem of needing to select different adapters according to different scenes is solved. There is no need to train and deploy target recognition models for the target detection requirements of each target scene, which can greatly shorten the research and development cycle of target recognition applications and reduce research and development and maintenance costs. The target recognition device disclosed in the embodiment of the present application is used to quickly launch multiple requirements based on a target recognition model, thereby solving the problems of fragmented requirements, complex algorithm maintenance, and high costs.

[0131] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0132] The above is a detailed introduction to a target identification method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and a core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0133] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0134] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the electronic device according to the embodiment of the present application. The present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0135] For example, Figure 8 An electronic device that can implement the method according to the present application is shown. The electronic device can be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc. The electronic device traditionally includes a processor 810 and a memory 820 and a program code 830 stored on the memory 820 and can be run on the processor 810, and the processor 810 implements the method described in the above embodiment when executing the program code 830. The memory 820 can be a computer program product or a computer-readable medium. The memory 820 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. The memory 820 has a storage space 8201 for the program code 830 of the computer program for executing any method step in the above method. For example, the storage space 8201 for the program code 830 can include various computer programs for implementing the various steps in the above method respectively. The program code 830 is a computer-readable code. These computer programs can be read from one or more computer program products or written into the one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. The computer program includes a computer readable code, and when the computer readable code is executed on an electronic device, the electronic device is caused to execute the method according to the above embodiment.

[0136] The embodiment of the present application further discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the target identification method described in the embodiment of the present application are implemented.

[0137] Such a computer program product may be a computer-readable storage medium, which may have a computer-readable storage medium having a Figure 8 The memory 820 in the electronic device shown in the figure is similar to the storage segment, storage space, etc. The program code can be compressed and stored in the computer-readable storage medium in an appropriate form. The computer-readable storage medium is generally as shown in the reference Fig. 9 The portable or fixed storage unit. Generally, the storage unit includes computer readable code 830', which is a code read by a processor. When the code is executed by the processor, each step in the method described above is implemented.

[0138] The term "one embodiment", "embodiment" or "one or more embodiments" herein means that a particular feature, structure or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present application. In addition, please note that the examples of the term "in one embodiment" here do not necessarily all refer to the same embodiment.

[0139] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0140] In the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A target recognition method, characterized in that: The method comprises: A preset target recognition model, the target recognition model includes: an image feature extraction module, a language feature extraction module and a target extraction module, wherein the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, and a first output module; Obtaining an image to be identified in a target scene and a prompt text indicating a target to be identified; Performing image feature extraction on the image to be recognized to obtain image features through one or more of the first expert modules that are automatically adapted to the target scene; and performing language feature extraction on the prompt text to obtain language features through the language feature extraction module; Performing multimodal feature extraction on the image features and the language features through one or more second expert modules that are automatically adapted to the target scene to obtain multimodal features; The first output module performs feature mapping on the multimodal features to obtain a target recognition result corresponding to the image to be recognized.

2. The method according to claim 1, characterized in that The first hybrid adapter multi-expert module also includes: a first routing module, the first expert modules are connected in parallel through the first routing module, and the one or more first expert modules that are automatically adapted to the target scene are used to extract image features from the image to be identified to obtain image features, including: Calling one or more of the first expert modules associated with the target scene through the first routing module to obtain single expert module output features output by the called first expert modules; The output features of the single expert module are integrated to obtain image features.

3. The method according to claim 1, characterized in that The target extraction module further includes: a second routing module, the second expert modules are connected in parallel through the second routing module, and the multimodal feature extraction is performed on the image feature and the language feature through one or more second expert modules that are automatically adapted to the target scene to obtain the multimodal feature, including: Calling one or more of the second expert modules associated with the target scene through the second routing module, performing multimodal feature extraction on the image features and the language features, and obtaining a single expert module multimodal output; The multimodal outputs of the single expert module are fused to obtain multimodal features.

4. The method according to claim 1, characterized in that: The target recognition result includes: a positioning frame, and after the first output module performs feature mapping on the multimodal feature and obtains the target recognition result corresponding to the image to be recognized, it also includes: In response to the target recognition result indicating that the image to be recognized includes a warning target, the image to be recognized is classified by using the pre-trained classification capability of the target recognition model to obtain a target category matched by the image to be recognized; Annotating the image to be identified based on the target category and the positioning frame to generate incremental sample data; Based on the incremental sample data, fine-tune the target recognition model.

5. The method according to claim 4, characterized in that The image feature extraction module includes: a first hybrid adapter multi-expert module and a first LoRA module, the target extraction module includes: a second hybrid adapter multi-expert module and a second LoRA module, and the fine-tuning of the target recognition model based on the incremental sample data includes: Freeze the model parameters of the target recognition model except the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module and the second LoRA module, and fine-tune the model parameters of the first hybrid adapter multi-expert module, the first LoRA module, the second hybrid adapter multi-expert module and the second LoRA module based on the incremental sample data.

6. The method according to claim 5, characterized in that The image feature extraction module includes multiple transformer structures, and the first hybrid adapter multi-expert module and the first LoRA module are located in the high-level transformer structure; the target extraction module includes multiple transformer structures, and the second hybrid adapter multi-expert module and the second LoRA module are located in each of the transformer structures.

7. The method according to claim 1, characterized in that The target recognition model is trained by the following method: Using a picture-text matching training method to train the image feature extraction module and the language feature extraction module; The model parameters of the image feature extraction module and the language feature extraction module obtained through training are frozen, and a multimodal target detection task is performed to train the target extraction module to obtain a target recognition model.

8. A target recognition device, characterized in that: The device comprises: The target recognition model setting unit is used to preset the target recognition model, wherein the target recognition model includes: an image feature extraction module, a language feature extraction module and a target extraction module, wherein the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, and the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, and a first output module; An input acquisition unit, used to acquire an image to be recognized in a target scene and a prompt text indicating a target to be recognized; A first feature extraction unit is used to extract image features from the image to be identified by using one or more of the first expert modules that are automatically adapted to the target scene to obtain image features; and to extract language features from the prompt text by using the language feature extraction module to obtain language features; a second feature extraction unit, configured to perform multimodal feature extraction on the image feature and the language feature by using one or more second expert modules automatically adapted to the target scene to obtain a multimodal feature; The target recognition unit is used to perform feature mapping on the multimodal features through the first output module to obtain a target recognition result corresponding to the image to be recognized.

9. A target recognition system, characterized in that: The system includes: a preset target recognition model, wherein the target recognition model includes: an image feature extraction module, a language feature extraction module and a target extraction module, wherein the image feature extraction module includes: a first hybrid adapter multi-expert module composed of a plurality of different first expert modules, the target extraction module includes: a second hybrid adapter multi-expert module composed of a plurality of different second expert modules, and a first output module, and the system also includes: An image acquisition unit, used for acquiring an image to be recognized in a target scene; A text collection unit, used to collect prompt text in the target scene; The target recognition result acquisition unit is used to use the image to be recognized and the prompt text as inputs of the target recognition model, call the target recognition model to execute the following target recognition steps, and obtain the target recognition result corresponding to the image to be recognized: extract image features of the image to be recognized through one or more of the first expert modules that are automatically adapted to the target scene to obtain image features; and extract language features of the prompt text through the language feature extraction module to obtain language features; extract multimodal features of the image features and the language features through one or more of the second expert modules that are automatically adapted to the target scene to obtain multimodal features; and perform feature mapping on the multimodal features through the first output module to obtain the target recognition result corresponding to the image to be recognized.

10. An electronic device comprising a memory, a processor, and a program code stored in the memory and executable on the processor, characterized in that: When the processor executes the program code, the method according to any one of claims 1 to 7 is implemented.

11. A computer-readable storage medium having a program code stored thereon, characterized in that: When the program code is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images