Multi-task model training method, prediction method, equipment, storage medium and product

Abstract

The invention discloses a multi-task model training method, a prediction method, equipment, a storage medium and a product. The training method comprises the following steps: performing feature extraction on an image of sample data by using an image feature extraction module; performing feature extraction on the text description of the sample data by using a text feature extraction module; aligning the image feature vector and the text feature vector, and adjusting parameters of an image feature extraction module and a text feature extraction module according to a matching relationship between the image feature vector and the text feature vector; parameters of the text feature extraction module are migrated to a feature fusion module; performing fusion and feature extraction on the prompt vector and the image feature vector by using a feature fusion module to obtain prediction text description; and adjusting parameters of a feature fusion module according to the predicted text description and the text description to obtain a multi-task model. According to the method, the problems of poor model performance and hardware resource waste caused by tedious processing of various sample data and scarcity of the sample data are solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and particularly relates to a multi-task model training method and prediction method for the security and protection field, equipment, storage medium and product. BACKGROUND

[0002] The development of the security and protection field has gone through the course from detection and alarm technology to access control and monitoring technology, from physical space security to digital space security, from analog security and protection to digital security and protection, and then to network security and intelligent security and protection. At present, it has fully entered the stage of intelligent security and protection, and the technology based on artificial intelligence has been fully applied in the security and protection field.

[0003] The existing algorithm based on artificial intelligence (AI) is applied more and more widely in the security and protection field, such as target classification, target detection, target tracking, image segmentation, image generation and restoration, image description generation, face recognition, behavior analysis, signal detection and recognition, radar and visual fusion, etc. For different downstream tasks in different scenes and different needs, the existing algorithm usually constructs and generates a specific AI small model for a certain specific downstream task, thereby obtaining different AI small models, and then a large number of experiments are carried out on each AI small model to realize optimization and upgrading in order to achieve good results. Finally, each type of AI small model based on different neural network architectures is deployed on different software and hardware platforms. This way of constructing and training the corresponding AI small model for each downstream task has the following problems:

[0004] (1) The AI small model for a specific task cannot be extended and expanded, and cannot complete other downstream tasks except the specific task.

[0005] (2) There are different downstream task needs in the security and protection field, and the corresponding AI small model needs to be constructed according to each downstream task. However, this way not only requires researchers to master the knowledge of one or more technical directions corresponding to various downstream tasks, but also needs a large number of experiments to optimize each AI small model, which not only increases the research difficulty, but also wastes time and effort, and requires multiple technical field researchers to participate, lacking uniformity.

[0006] (3) When constructing corresponding AI small models for each downstream task, a certain number of sample data corresponding to the AI small models need to be constructed for training the AI small models. However, the sample data generally come from different scenes or different needs, and different methods need to be taken for preprocessing and labeling, so that the data processing process is extremely complicated. Meanwhile, in some cases, the sample data of some downstream tasks may be difficult to collect in large quantities, which may also lead to poor performance of the AI small models of these downstream tasks, and different deployment of AI small models may take different neural network architectures and need different hardware support, which may further lead to waste of resources. SUMMARY

[0007] The present application aims to provide a multi-task model training method, a prediction method, an equipment, a storage medium and a product, to solve the problem that in the existing security field, multiple AI small models are constructed to solve multiple downstream tasks, which leads to the inability to extend and expand between multiple AI small models, the need for time-consuming and labor-intensive targeted research and optimization, the complexity of preprocessing various sample data, the possible sample data scarcity leading to model performance bottleneck, and the possible hardware resource waste in multi-model deployment.

[0008] The present application solves the above technical problems through the following technical solutions: a multi-task model training method, the training method comprising the following steps:

[0009] constructing a basic sample data set, the basic sample data set comprising at least a security general scene data set, each sample data in the basic sample data set comprising an image and a text description thereof;

[0010] constructing an artificial intelligence large model, the artificial intelligence large model comprising an image feature extraction module, a text feature extraction module, a feature alignment module and a feature fusion module;

[0011] training the artificial intelligence large model using the basic sample data set, the specific training process comprising a first training phase and a second training phase; the first training phase comprising:

[0012] using the image feature extraction module to perform image feature extraction on the image of each sample data in the basic sample data set to obtain an image feature vector; using the text feature extraction module to perform text feature extraction on the text description of each sample data in the basic sample data set to obtain a text feature vector; using the feature alignment module to align the image feature vector and the text feature vector, and adjusting the parameters of the image feature extraction module and the text feature extraction module according to the matching relationship of the aligned image feature vector and text feature vector;

[0013] the second training phase comprising:

[0014] migrate the parameters of the text feature extraction module to the feature fusion module in a shared manner;

[0015] fuse and extract features of the obtained prompt vector and the image feature vector by using the feature fusion module to obtain a predicted text description; adjust the parameters of the feature fusion module according to the predicted text description and the text description corresponding to the image feature vector to obtain a multi-task model; wherein the prompt vector is empty or a word sequence vector converted from a descriptive question.

[0016] Further, the specific construction process of the basic sample data set is:

[0017] download the required open source data set through the network; parse each open source data set according to the compression rules published by the open source data set to obtain a large sample data set;

[0018] collect and organize general security scene images, and perform text description on the general security scene images to obtain the general security scene images and their text descriptions; construct a general security scene data set according to the general security scene images and their text descriptions;

[0019] integrate the large sample data set and the general security scene data set to obtain the basic sample data set.

[0020] Further, the image feature extraction module includes a dynamic sequence generator and an image encoder, the dynamic sequence generator is used to process the image of each sample data in the basic sample data set to obtain a dynamic sequence; and the image encoder is used to extract features of the dynamic sequence to obtain the image feature vector.

[0021] Further, the dynamic sequence generator is specifically used for:

[0022] cropping or scaling the image of each sample data in the basic sample data set to obtain an image of a fixed size;

[0023] slicing and rearranging the image of a fixed size to obtain an image sequence composed of a plurality of sub-image blocks;

[0024] extracting the information amount of the image sequence to obtain an information amount binary mask of each sub-image block in the image sequence;

[0025] select the sub-image blocks with high information amount and the sub-image blocks with low information amount according to the information amount binary mask of each sub-image block, and reorganize the sub-image blocks with high information amount and the sub-image blocks with low information amount respectively to obtain a high information amount image sequence and a low information amount image sequence;

[0026] cross-fuse the high-information image sequence and the low-information image sequence to obtain a fused image sequence;

[0027] randomly generate a first position encoding, multiply or add the first position encoding and the fused image sequence to obtain a dynamic sequence.

[0028] Further, an image information distribution network stacked by multiple multi-layer perceptrons is used to extract information of the image sequence to obtain an information quantity binary mask of each sub-image block in the image sequence.

[0029] Further, the high-information image sequence and the low-information image sequence are cross-fused, and the specific implementation process is as follows:

[0030] a first fully connected layer is used to integrate and compress the high-information image sequence to obtain a query vector of the high-information image sequence;

[0031] a second fully connected layer is used to integrate and compress the low-information image sequence to obtain a key vector of the low-information image sequence;

[0032] a third fully connected layer is used to integrate and compress the low-information image sequence to obtain a value vector of the low-information image sequence;

[0033] a dot product is calculated between the query vector of the high-information image sequence and the key vector of the low-information image sequence, and a Softmax operation is performed on the dot product result to obtain a weight matrix;

[0034] the weight matrix is multiplied by the value vector of the low-information image sequence to obtain a fused image sequence.

[0035] Further, the image encoder is an image encoder based on a Transformer architecture, and the image encoder is stacked by L1 self-attention layers + a feedforward network.

[0036] Further, the text feature extraction module is specifically used for:

[0037] word segmentation is performed on a text description of each sample data in the basic sample data set to obtain a word sequence composed of multiple words;

[0038] each word in the word sequence is converted into a word vector;

[0039] a second position encoding is randomly generated for each word vector, and the second position encoding is multiplied or added to the word vector to obtain a word sequence vector;

[0040] The text encoder stacked by L2 self-attention layers and a feedforward network is used to extract features of the word sequence vector, and a text feature vector is obtained.

[0041] Further, the feature alignment module is specifically configured to:

[0042] The image feature vector and the text feature vector are respectively subjected to dimension transformation, so that the image feature vector and the text feature vector are mapped to the same dimension, and an aligned image feature vector and an aligned text feature vector are obtained.

[0043] The similarity of the aligned image feature vector and the aligned text feature vector is calculated, and the parameters of the image feature extraction module and the text feature extraction module are adjusted according to the matching relationship between the aligned image feature vector and the aligned text feature vector, so that the similarity of the aligned image feature vector and the aligned text feature vector is maximized or minimized.

[0044] The aligned image feature vector and the aligned text feature vector are respectively integrated and compressed, and a first query vector and a second query vector are obtained. The mean square error of the first query vector and the second query vector is calculated, and the parameters of the image feature extraction module and the text feature extraction module are adjusted according to the matching relationship between the first query vector and the second query vector, so that the mean square error of the first query vector and the second query vector is maximized or minimized.

[0045] Further, the feature fusion module comprises a text decoder and a multi-task head network, the text decoder is used to fuse the obtained prompt vector and the image feature vector, and an image-based text feature is obtained.

[0046] The multi-task head network is used to extract features of the image-based text feature, and a predicted text description is obtained.

[0047] Further, the multi-task head network adopts a large language model based on a Transformer architecture, and the large language model is stacked by L3 self-attention layers, cross-attention layers and a feedforward network.

[0048] Further, the first training stage and the second training stage are independently performed or alternately performed by using a round-robin strategy.

[0049] Based on the same concept, the application further provides a multi-task prediction method, which comprises the following steps:

[0050] Task sample data sets are constructed according to prediction tasks, and each sample data in the task sample data set comprises an image and a manually annotated label thereof.

[0051] The multi-task model trained by the multi-task model training method is called, and parameters of a feature fusion module in the multi-task model are fine-tuned by using the task sample data set, parameters of an image feature extraction module and a text feature extraction module remain unchanged, and a target task model is obtained.

[0052] A to-be-predicted image is obtained, and the target task model is used to predict the to-be-predicted image to obtain a prediction result.

[0053] Based on the same concept, the application further provides a terminal device, comprising a memory, a processor, and a computer program / instruction stored in the memory, wherein the processor executes the computer program / instruction to implement the method described above.

[0054] Based on the same concept, the application further provides a computer-readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to implement the method described above.

[0055] Based on the same concept, the application further provides a computer program product, which comprises a computer program / instruction, and the computer program / instruction is executed by a processor to implement the method described above.

[0056] Advantages

[0057] Compared with the prior art, the application has the following advantages:

[0058] The application is based on the security field, and proposes an AI large model construction and training method that can be used for multiple tasks. In the training stage, the AI large model can be trained based on a large amount of open source data sets or a large amount of general task data that is easy to collect, to obtain a multi-task model. Before specific task prediction, the multi-task model is fine-tuned based on a small amount of data of one or more specific tasks, to obtain a target task model for the specific task prediction, thereby realizing incremental training and optimization of the multi-task model and improving the performance of the target task model. The application realizes image description generation and image-text matching based on images and text descriptions, solves multiple downstream tasks such as pedestrian detection, vehicle detection, and behavior analysis through one AI large model, and realizes extension and expansion of one AI large model among different tasks.

[0059] The application completes multiple downstream tasks through one AI large model, without the need for researchers to master theoretical knowledge of multiple technical directions. For small models of various specific scenarios, it is not necessary to perform model construction from beginning to end and tedious data preprocessing and optimization experiments, but only a small amount of task sample data is needed to perform simple incremental fine-tuning on the AI large model, to achieve the same or even better effect, thereby reducing the consumption of human resources.

[0060] The multi-task model is deployed to a corresponding hardware platform, a model solves multiple tasks, avoids the problem of poor model performance caused by complex multi-task sample data processing or insufficient sample data, and facilitates model deployment, and avoids resource waste caused by multiple models requiring multiple hardware platforms. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only one embodiment of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0062] Figure 1 is a multi-task model training method flowchart in the embodiment of the present application;

[0063] Figure 2 is an artificial intelligence large model architecture diagram in the embodiment of the present application;

[0064] Figure 3 is a multi-task prediction method flowchart in the embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0066] The technical solutions of the present application will be described in detail in the following specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

[0067] In the security field, the existing artificial intelligence-based method mainly completes and solves various sub-tasks with different needs in different scenarios by constructing different AI small models. However, this method also brings some problems, that is, a certain amount of sample data is needed for different tasks and different algorithm models to obtain a better algorithm model, and a large number of tuning experiments need to be performed according to different algorithm models. These make the data preprocessing process and the algorithm model optimization process not only time-consuming and laborious, but also extremely complex. On the other hand, the sample data in some scenarios may be scarce, which will also limit the performance of the algorithm model, causing a performance bottleneck, and the deployment of different models may adopt different neural network architectures, which need different hardware support, causing resource waste, and will also be limited in actual use.

[0068] Based on the above technical problems, the present invention provides a multi-task model training method, prediction method, equipment, storage medium and product. Through AI big model technology, an AI big model that can realize multiple tasks is constructed. Researchers do not need to master theoretical knowledge of multiple technical directions, do not need to rebuild small models for each specific scene and conduct tedious tuning experiments, and do not need to perform uncontrollable preprocessing of various types of sample data, thereby avoiding problems such as model performance bottlenecks caused by possible scarcity of sample data and possible waste of hardware resources in multi-model deployment. By inputting video or image into the AI ​​big model, the target of interest in the video or image and its behavior, appearance and other attribute features can be obtained. By inputting text into the big model, the corresponding scene can be output in massive historical videos or real-time videos for backtracking and monitoring, ultimately achieving effective security protection and control.

[0069] Example 1

[0070] like Figure 1 As shown, a multi-task model training method provided by an embodiment of the present invention includes the following steps:

[0071] Step S1: constructing a basic sample data set, wherein the basic sample data set includes at least a general security scene data set, and each sample data in the basic sample data set includes an image and its text description;

[0072] Step S2: constructing an artificial intelligence large model, wherein the artificial intelligence large model includes an image feature extraction module, a text feature extraction module, a feature alignment module, and a feature fusion module;

[0073] Step S3: Use the basic sample data set to train the artificial intelligence large model to obtain a multi-task model.

[0074] In the embodiment of the present invention, the specific process of constructing the basic sample data set is as follows:

[0075] Download the required open source datasets from the Internet; parse them according to the compression rules published by each open source dataset to obtain a large sample dataset;

[0076] Collect and organize general security scene images, and perform text descriptions on the general security scene images to obtain general security scene images and their text descriptions; construct a general security scene dataset based on the general security scene images and their text descriptions;

[0077] The large sample dataset is integrated with the general security scenario dataset to obtain the basic sample dataset.

[0078] The open source dataset is an existing dataset, and the dataset often has image data or radar data and corresponding label information (i.e., text description). Without labeling the data in the open source dataset, the complex data preprocessing process is avoided. Meanwhile, the data in the open source dataset is rich, which avoids the performance bottleneck of the model caused by the scarcity of sample data, and ensures the performance of the trained multi-task model. In the embodiment of the present application, the downloaded open source dataset includes VOC dataset, COCO dataset and ImageNet dataset for target detection, KITTI dataset, NuScenes dataset and BDD100K dataset in the field of autonomous driving, and image-text dataset, such as SUN09 dataset focusing on scene understanding, HICO-DET dataset focusing on human interaction detection, and LAION-5B dataset with image-text pairs with a capacity of more than 80T.

[0079] The general security scene includes prison environment, border environment, various boundaries and perimeter environment, and common road, grassland, jungle, desert and other environments. The general security scene image needs to be collected manually and labeled by artificial. The label is a text description or a text description and a target detection box.

[0080] Figure 2 An artificial intelligence large model architecture diagram is shown. The artificial intelligence large model includes an image feature extraction module, a text feature extraction module, a feature alignment module and a feature fusion module. The image feature extraction module includes a dynamic sequence generator and an image encoder. The text feature extraction module includes a text encoder. The feature fusion module includes a text decoder and a multi-task head network. The dynamic sequence generator is used to process the input image to obtain a dynamic sequence. The image encoder is used to extract features from the dynamic sequence to obtain an image feature vector. The text encoder is used to extract features from the input text description to obtain a text feature vector. The feature alignment module is used to align the image feature vector and the text feature vector, and adjust the parameters of the image encoder and the text encoder according to the matching relationship of the aligned image feature vector and the text feature vector, i.e., the training of the image encoder and the text encoder is realized. The text decoder is used to fuse the input prompt vector and the image feature vector to obtain an image-based text feature. The multi-task head network is used to extract features from the image-based text feature to obtain a predicted text description, and adjust the parameters of the text decoder or the text decoder and the multi-task head network according to the predicted text description and the text description corresponding to the image feature vector, i.e., the training of the text decoder or the text decoder and the multi-task head network is realized.

[0081] The training of the artificial intelligence large model by using the basic sample data set comprises a first training stage and a second training stage, and the first training stage and the second training stage can be independently trained or alternately trained by adopting a round-robin strategy. When the independent training mode is adopted, i.e. the parameters of the image encoder and the text encoder are trained in the first training stage, the parameters of the text decoder or the text decoder and the multi-task head network are in an initialization state (or a frozen state) and do not participate in the training in the first training stage; when the parameters of the text decoder or the text decoder and the multi-task head network are trained in the second training stage, the parameters of the image encoder and the text encoder remain unchanged and do not participate in the training in the second training stage, and the parameters of the text encoder are used to initialize the text decoder. When the alternately training mode by adopting the round-robin strategy is adopted, i.e. the first training stage is trained for T1 times, then the second training stage is trained for T2 times, then the first training stage is trained for T3 times, then the second training stage is trained for T4 times, and the training is alternately performed in turn until a set number of training times is reached; when the alternately training mode by adopting the round-robin strategy is adopted, the parameters (i.e. L2 self-attention layer parameters and feedforward network parameters) of the same network architecture part of the text encoder and the text decoder are shared.

[0082] That is, the parameters of the text encoder and the text decoder are shared and jointly trained and optimized.

[0083] In the embodiment of the present application, the first training stage comprises:

[0084] Step A1: performing image feature extraction on the image of each sample data in the basic sample data set by using an image feature extraction module to obtain an image feature vector;

[0085] Step A2: performing text feature extraction on the text description of each sample data in the basic sample data set by using a text feature extraction module to obtain a text feature vector;

[0086] Step A3: aligning the image feature vector and the text feature vector by using a feature alignment module, and adjusting the parameters of the image feature extraction module and the text feature extraction module according to the matching relationship of the aligned image feature vector and text feature vector.

[0087] The specific implementation process of step A1 is: processing the image of each sample data in the basic sample data set by using a dynamic sequence generator to obtain a dynamic sequence; and performing feature extraction on the dynamic sequence by using an image encoder to obtain an image feature vector. The specific implementation process of processing the image of each sample data in the basic sample data set by using the dynamic sequence generator is:

[0088] The image of each sample data in the basic sample data set is cropped or scaled to obtain an image with a fixed size;

[0089] slicing and rearranging the fixed-size images to obtain an image sequence composed of multiple sub-image blocks;

[0090] extracting information quantity from the image sequence to obtain an information quantity binary mask of each sub-image block in the image sequence;

[0091] selecting the sub-image blocks with high information quantity and the sub-image blocks with low information quantity according to the information quantity binary masks of the sub-image blocks, and recombining the sub-image blocks with high information quantity and the sub-image blocks with low information quantity respectively to obtain a high-information-quantity image sequence and a low-information-quantity image sequence;

[0092] cross-fusing the high-information-quantity image sequence and the low-information-quantity image sequence to obtain a fused image sequence;

[0093] generating a first position code randomly, multiplying or adding the first position code with the fused image sequence to obtain a dynamic sequence.

[0094] In order to unify the input image size, the images of each sample data in the basic sample data set are cropped or scaled, and the size of the obtained images is fixed as WxH, wherein W is the image width and H is the image height; the fixed-size images are sliced to obtain N W1xH1 sub-image blocks, and the N sub-image blocks are randomly rearranged, that is, an image sequence with a size of NxW1xH1 is obtained. Each sub-image block in the image sequence is derived from the original image and does not overlap, wherein N=(W / W1)*(H / H1).

[0095] In the embodiment of the application, the image information quantity distribution network stacked (i.e., connected in sequence) by multiple multilayer perceptrons (MLP) is used to extract information quantity from the image sequence to obtain an information quantity binary mask of each sub-image block in the image sequence. The information quantity binary mask of each sub-image block is obtained by binarizing the weight coefficient of each sub-image block, that is, the information quantity binary mask of each sub-image block is 1 or 0. The importance of the sub-image block is evaluated through the information quantity binary mask of each sub-image block. The sub-image block with the information quantity binary mask of 1 is the sub-image block with high information quantity, and the sub-image block with the information quantity binary mask of 0 is the sub-image block with low information quantity, thereby obtaining a high-information-quantity image sequence and a low-information-quantity image sequence.

[0096] The size of the high-information image sequence is N1xW1xH1, where N1 is the number of sub-image blocks in the high-information image sequence, and the high-information image sequence sufficiently represents the semantic information of the entire image. Meanwhile, instead of being discarded directly, the low-information image sequence is taken as an input together with the high-information image sequence, and the two are input into an image sequence merging network stacked by a cross-attention layer (Cross-Attention Layer) and a feed forward network (FFN), so as to fuse or compress the low-information sub-image blocks into the high-information image sequence, and finally realize dynamic conversion of the number of sub-image blocks in the image sequence from N to N1.

[0097] In the embodiment of the application, the high-information image sequence and the low-information image sequence are cross-fused, and the specific implementation process is as follows:

[0098] For the high-information image sequence, a full connection layer (i.e., a first full connection layer) is used to obtain a query (Query) vector of the high-information image sequence;

[0099] For the low-information image sequence, two different full connection layers (i.e., a second full connection layer and a third full connection layer) are used to obtain a key (Key) vector and a value (Value) vector of the low-information image sequence, respectively;

[0100] Dot product is calculated between the query vector of the high-information image sequence and the key vector of the low-information image sequence, and a Softmax operation is performed on the dot product result to obtain a weight matrix;

[0101] The weight matrix is multiplied by the value vector of the low-information image sequence to obtain a fused image sequence with a size of N1xW1xH1.

[0102] Compared with the image sequence, the number of sub-image blocks in the fused image sequence is reduced, the redundancy between the image sequences is reduced, the training efficiency is improved, and a large amount of redundant calculation is avoided. Meanwhile, since each sub-image block in the fused image sequence comes from a different position of the original image, in order to prevent the spatial characteristics from being destroyed and causing confusion of high-level semantic features, and thus affecting the performance of the model, a random position encoding (i.e., a first position encoding) with a size of N1x1 is assigned to the fused image sequence, and finally a dynamic sequence of input images is formed.

[0103] In the embodiment of the application, the image encoder is a Transformer architecture-based image encoder, and the image encoder is stacked by L1 self-attention layers and a feed forward network, i.e., the image encoder includes L1 layers, and each layer includes a self-attention layer and a feed forward network.

[0104] The specific implementation process of step A2 is: performing word segmentation on the text description of each sample data in the basic sample data set to obtain a word sequence composed of multiple words; converting each word in the word sequence into a word vector; randomly generating a second position encoding for each word vector, and multiplying or adding the second position encoding and the word vector to obtain a word sequence vector; and performing feature extraction on the word sequence vector by using a text encoder stacked by L2 self-attention layers and a feedforward network to obtain a text feature vector.

[0105] For example, the text description "I am walking in the park" is segmented by words to obtain a word sequence composed of the words "I", "am", "walking", and "in the park". Similarly, since each word vector comes from a different position of the original text description, in order to prevent the spatial characteristics from being destroyed and causing confusion of high-level semantic features, thereby affecting the performance of the model, a random position encoding (i.e., a second position encoding) that can be automatically learned during training is assigned to each word vector, and finally a word sequence vector of the input text description is formed.

[0106] In the embodiment of the present application, the text encoder is a text encoder based on the Transformer architecture, and the text encoder is stacked by L2 self-attention layers and a feedforward network, that is, the text encoder includes L2 layers, and each layer includes a self-attention layer and a feedforward network.

[0107] The specific implementation process of step A3 is:

[0108] Step A3.1: constructing a dimension alignment network based on an MLP, and performing dimension transformation on the image feature vector output by the image encoder and the text feature vector output by the text encoder through the dimension alignment network, so that the image feature vector and the text feature vector are mapped to the same dimension to obtain an aligned image feature vector and an aligned text feature vector;

[0109] Step A3.2: calculating the similarity of the aligned image feature vector and the aligned text feature vector, and adjusting the parameters of the image feature extraction module and the text feature extraction module according to the matching relationship between the aligned image feature vector and the aligned text feature vector, so as to maximize or minimize the similarity of the aligned image feature vector and the aligned text feature vector.

[0110] Step A3.3: integrating and compressing the aligned image feature vector and the aligned text feature vector respectively to obtain a first query vector and a second query vector; calculating the mean square error of the first query vector and the second query vector, and adjusting the parameters of the image feature extraction module and the text feature extraction module according to the matching relationship between the first query vector and the second query vector, so as to maximize or minimize the mean square error of the first query vector and the second query vector.

[0111] Steps A3.2 and A3.3 can be carried out simultaneously. During the training process, the image is input into the image feature extraction module, and the text description corresponding to the image is input into the text feature extraction module, so as to match the image feature vector and the text feature vector obtained thereby; if a plurality of images numbered 1-n are input into the image feature extraction module at one time, a plurality of text descriptions numbered 1-n are input into the text feature extraction module, and the images and the text descriptions numbered the same (for example, the image numbered 1 corresponds to the text description numbered 1) correspond to each other, then the image feature vector numbered 1 is matched with the text feature vector numbered 1, and the image feature vector numbered 1 is not matched with the text feature vectors numbered 2-n.

[0112] In step A3.2, the similarity of the aligned image feature vector and the text feature vector is calculated by using the cosine similarity calculation, if the aligned image feature vector and the text feature vector match, the parameters of the image encoder and the text encoder are adjusted to maximize the similarity of the aligned image feature vector and the text feature vector; if the aligned image feature vector and the text feature vector do not match, the parameters of the image encoder and the text encoder are adjusted to minimize the similarity of the aligned image feature vector and the text feature vector. In another specific embodiment of the present application, only the matched image feature vector and the text feature vector can be aligned to obtain a text-image matching pair, and then the parameters of the image encoder and the text encoder are adjusted to maximize the similarity of the text-image matching pair.

[0113] In step A3.3, the image feature vector is passed through a fully connected layer to obtain a first query vector of the image feature vector; the text feature vector is passed through another fully connected layer to obtain a second query vector of the text feature vector; then the mean square error of the first query vector and the second query vector is calculated, if the image feature vector corresponding to the first query vector and the text feature vector corresponding to the second query vector match, the parameters of the image encoder and the text encoder are adjusted to minimize the mean square error of the first query vector and the second query vector, that is, to make the matching degree of the image feature vector and the text feature vector higher; if the image feature vector corresponding to the first query vector and the text feature vector corresponding to the second query vector do not match, the parameters of the image encoder and the text encoder are adjusted to maximize the mean square error of the first query vector and the second query vector. In another specific embodiment of the present application, only the matched image feature vector and the text feature vector can be used to calculate the mean square error of the first query vector and the second query vector, and then the parameters of the image encoder and the text encoder are adjusted to minimize the mean square error of the first query vector and the second query vector.

[0114] The parameters of the image encoder and the text encoder are adjusted through a large amount of sample data of a basic sample data set, so that the performance of the image encoder and the text encoder in extracting features is greatly improved. After training in the first training stage, the image feature extraction module can better extract image features and obtain image high-level semantic information; and part of the parameters of the text feature extraction module can be shared with the text decoder.

[0115] In the embodiment of the present application, the second training stage comprises:

[0116] Step B1: The parameters of the text feature extraction module are transferred to the text decoder in the feature fusion module in a shared manner;

[0117] Step B2: The obtained prompt vector and the image feature vector are fused and feature-extracted by the feature fusion module to obtain a predicted text description;

[0118] Step B3: The parameters of the feature fusion module are adjusted according to the predicted text description and the text description corresponding to the image feature vector, to obtain a multi-task model; wherein the prompt vector is empty or a word sequence vector converted from a descriptive question.

[0119] In the embodiment of the present application, the image-based text decoder is a text decoder based on a Transformer architecture, and the text decoder is stacked by L2 self-attention layers + cross-attention layers + feedforward networks, that is, the text decoder includes L2 layers, each layer includes a self-attention layer, a cross-attention layer and a feedforward network connected in sequence. Considering feature reuse and reducing the number of parameters, the L2 self-attention layer parameters and the feedforward network parameters of the text encoder of the text feature extraction module are shared with the text decoder.

[0120] The text decoder is used for fusing the obtained prompt vector and the image feature vector to obtain image-based text features. The text decoder has two inputs: one is the image feature vector output by the image encoder, which is fused into the text decoder through a cross-attention layer; the other is a prompt vector, which can be set to be empty, that is, filled with a certain specific value, for example, the prompt vector is set to be all zero values; or it can be set to be a word sequence vector converted from a descriptive question, for example, the prompt vector is set to be a word sequence vector converted from the descriptive question "What is this picture of what category?". In actual training, the prompt vector is different according to different data sets, for example, some data sets include images and their non-dialogue text descriptions, at this time the prompt vector is set to be empty; some data sets include images and their dialogue text descriptions, that is, text questions are given based on image content and answers are given (for example, what is this picture? This picture is a dog), at this time the prompt vector is set to be a word sequence vector converted from a descriptive question.

[0121] The multi-task head network is used for feature extraction on the image-based text features output by the text decoder to obtain a predicted text description. In an embodiment of the present application, the multi-task head network adopts a large language model (LLM) constructed based on a Transformer architecture, and the large language model is stacked by L3 self-attention layers + cross-attention layers + feedforward networks, that is, the large language model includes L3 layers, and each layer includes a self-attention layer, a cross-attention layer and a feedforward network connected in turn.

[0122] It should be noted that the large language model can adopt a published and trained LLM model, such as LLaMA, OPT and the like. The LLM model has a large number of parameters, and it is time-consuming and laborious to re-initialize and then train it, so the idea of transfer learning can be used to initialize and freeze the parameters of the multi-task head network of the present application by using the model parameters of the LLM model trained by the published LLM model, and then the text features output by the text decoder are transformed through the network layer constructed by the multi-layer MLP to construct an input suitable for the selected LLM model at this time, that is, a soft input. Assuming that the size of the original input of the published LLM model is M11xM12, and the size of the text features output by the text decoder is M21xM22, the text features output by the text decoder cannot be directly connected to the LLM model at this time, so a network layer needs to be constructed for transformation. When the large language model adopts a published and trained model, the second training phase mainly trains the parameters of the text decoder.

[0123] The artificial intelligence large model is trained through the basic sample data set, the feature extraction capability of the large model is continuously improved, the implicit features in the massive data are better extracted, and a strong feature learning capability is laid for subsequent implementation of each specific task; through massive data and large-scale training, a pre-training large model (i.e., a multi-task model) with strong feature extraction and representation capabilities is finally obtained, and it is used as a basic large model.

[0124] Embodiment 2

[0125] As shown in Figure 2 A multi-task prediction method provided by an embodiment of the present application includes the following steps:

[0126] Step S1: constructing a task sample data set according to a prediction task, each sample data in the task sample data set including an image and a label artificially labeled by a person;

[0127] Step S2: calling the multi-task model trained by the multi-task model training method described in Embodiment 1, fine-tuning the parameters of the feature fusion module in the multi-task model using the task sample data set, and keeping the parameters of the image feature extraction module and the text feature extraction module unchanged to obtain a target task model.

[0128] Step S3: obtaining a to-be-predicted image, and predicting the to-be-predicted image using the target task model to obtain a prediction result.

[0129] Before performing prediction of a certain specific task, a sample data set of the specific task is first constructed, and the base large model (i.e., the multi-task model) obtained in Embodiment 1 is fine-tuned through the sample data set to adapt to prediction of the specific task.

[0130] In the target detection task, data of a specific environment and specific requirement is collected, such as picture or video data of pedestrians, vehicles, and animals in the environment of grassland, barbed wire, and fence in the airport perimeter, from which a target detection task sample data set is constructed; in the behavior analysis task, picture or video data of behaviors such as crossing the fence, throwing foreign objects, falling into the water, and leaving the post of patrol in a specific area is collected, from which a behavior analysis task sample data set is constructed; in the attribute recognition task, picture or video data such as appearance features, dressing features, face recognition, and crowd statistics is collected, from which an attribute recognition task sample data set is constructed; and in the radar vision fusion task, radar sensor data and corresponding camera video data are collected.

[0131] After the sample data of the prediction task is collected, manual labeling is performed, and different labels are labeled according to different tasks. For example, for the target detection task, the label is a text description of the position and category or mask of the target in the picture; for the picture-text description task, the label is a text description of the picture, such as a picture category description, a target behavior description in the picture, and an attribute description.

[0132] Due to environmental limitations or real sample distribution limitations, it is difficult to collect task sample data, or the probability of occurrence of a certain category or label in a real environment is relatively small. Therefore, in order to complete such a task, only a small amount of sample data based on these specific environments is needed to perform small-scale retraining and fine-tuning on the base large model (i.e., the multi-task model) in Embodiment 1, so that the feature learning is more suitable for the downstream tasks in the security field, and finally a multi-task large model is evolved.

[0133] In the fine-tuning in step S2, two fine-tuning methods can be combined, i.e., the prompt fine-tuning and the image-text locking method. In the prompt fine-tuning, the label corresponding to the image is adjusted, and the trainable prompt vector is fine-tuned. The specific implementation process is as follows: a term description belonging to a specific task is added to the text description, or a fixed description sentence is selectively added according to the specific task, so as to fine-tune the prompt vector and the text decoder. For example, for the image classification task, the classification category vocabulary is added, and a fixed sentence similar to "this is a 'cat' image" is created; for the behavior recognition task, the behavior category vocabulary is added, and a fixed sentence similar to "this is an image of a 'person' walking" is created; for the attribute recognition task, the attribute category vocabulary is added, and a fixed sentence similar to "this is an image of a 'female' 'person' walking" is created. In actual situations, simple or complex text descriptions can be created according to specific tasks, and then the multi-task model is incrementally trained through the task sample data set.

[0134] The image-text locking means that the model parameters are selectively locked and frozen during the fine-tuning. The selection basis comes from the actual situation. For example, for the image description task, the selected LLM model parameters are large and the re-initialization training is time-consuming and laborious, so the LLM model parameters can be locked and frozen. For example, because the image feature extraction module has a large number of parameters and the image encoder extraction performance is prevented from degrading due to fine-tuning, the image encoder parameters are locked and frozen, and so on.

[0135] The fine-tuning mainly performs the second training phase without the first training phase, so the fine-tuning mainly adjusts the parameters of the text decoder.

[0136] In step S3, when performing task prediction, the video or continuous multiple images and the prompt vector are taken as the input of the target task model, and finally the text description of the image is output. The text description content includes whether there is a target of interest, such as a pedestrian, a vehicle, an animal of a certain type, and other objects of interest. If there is a target, further questions can be asked, and the attribute information of the target can be output, such as the gender, age, height, fat, full-body clothing, face recognition, and comparison with the database to determine whether it is a registered person, and output the name and other pre-registered information, etc. For the vehicle target, the vehicle model, vehicle size, vehicle color, license plate recognition, etc. are output.

[0137] In the task prediction, the other application can also be: for continuous multiple image inputs, the text description content can include the behavior information of the target, such as the behaviors of personnel such as wandering, running, climbing, colliding, falling, crawling, or the behaviors of vehicles such as starting, parking, driving, braking, colliding, etc.; for continuous multiple video inputs, the text description content can include the spatial information of the target, such as the appearance of personnel or vehicles in multiple cameras, then the re-identification and feature matching are performed, the action trajectory of the target in all monitoring areas is described, and then the situation analysis is performed; for the input of other modal data such as radar sensor data, the text description content can also include the position information of the target, such as direction, distance, speed, altitude, etc., and then the positioning, navigation or real-time monitoring of the target in the monitoring area are performed by comprehensively integrating the information.

[0138] By implementing and outputting multiple tasks based on a large model, the functions of many small models are completed or covered, such as target recognition and classification, target detection, behavior analysis, face recognition, target re-identification, feature matching, multi-modal data fusion and other technologies, and then each downstream task corresponding to each artificial intelligence technology can select a suitable solution under the large model, thereby improving the overall performance of the security system.

[0139] Embodiment 3

[0140] The embodiment of the application also provides a terminal device, which comprises a memory, a processor and a computer program / instruction stored in the memory, and the processor executes the computer program / instruction to implement the method described in Embodiment 1 or Embodiment 2.

[0141] Although not shown, the terminal device comprises a processor, which can perform various appropriate operations and processes according to programs and / or data stored in a read-only memory (ROM) or programs and / or data loaded from a storage part into a random access memory (RAM). The processor can be a multi-core processor or can comprise multiple processors. In some embodiments, the processor can comprise a general-purpose main processor and one or more special-purpose coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processing unit (NPU), a digital signal processor (DSP) and the like. In the RAM, various programs and data required for the operation of the terminal device are also stored. The processor, the ROM and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0142] The above processor and memory are used together to execute programs / instructions stored in the memory, which can implement the methods, steps or functions described in the above embodiments when executed by a computer.

[0143] Although not shown, the embodiments of the present application also provide a computer readable storage medium having stored thereon computer programs / instructions which, when executed by a processor, implement the method of Embodiment 1 or Embodiment 2.

[0144] The storage medium of the embodiments of the present application includes permanent and non-permanent, removable and non-removable, and information storage can be realized by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0145] The storage medium includes permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carriers.

[0146] Although not shown, the embodiments of the present application also provide a computer program product, comprising: computer programs / instructions which, when executed by a processor, implement the method as described in Embodiment 1 or Embodiment 2.

[0147] The present application is supported by Hunan Provincial Science and Technology Innovation Plan, project number: 2023RC3230.

[0148] The above disclosure is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or modifications within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A multi-task model training method, characterized in that: The training method comprises the following steps: Constructing a basic sample data set, wherein the basic sample data set includes at least a general security scene data set, and each sample data in the basic sample data set includes an image and a text description thereof; Constructing an artificial intelligence large model, which includes an image feature extraction module, a text feature extraction module, a feature alignment module, and a feature fusion module; The artificial intelligence model is trained using the basic sample data set. The specific training process includes a first training phase and a second training phase. The first training phase includes: Using the image feature extraction module to extract image features from the image of each sample data in the basic sample data set to obtain an image feature vector; using the text feature extraction module to extract text features from the text description of each sample data in the basic sample data set to obtain a text feature vector; using the feature alignment module to align the image feature vector and the text feature vector, and adjusting the parameters of the image feature extraction module and the text feature extraction module according to the matching relationship between the aligned image feature vector and the text feature vector; The second training phase includes: Migrating the parameters of the text feature extraction module to the feature fusion module in a shared manner; The obtained prompt vector and the image feature vector are fused and feature extracted by using the feature fusion module to obtain a predicted text description; the parameters of the feature fusion module are adjusted according to the predicted text description and the text description corresponding to the image feature vector to obtain a multi-task model; wherein the prompt vector is empty or a word sequence vector converted from a descriptive question.

2. The multi-task model training method according to claim 1, characterized in that: The specific construction process of the basic sample data set is as follows: Download the required open source datasets from the Internet; parse them according to the compression rules published by each open source dataset to obtain a large sample dataset; Collect and organize general security scene images, and perform text descriptions on the general security scene images to obtain general security scene images and their text descriptions; construct a general security scene dataset based on the general security scene images and their text descriptions; The large sample data set is integrated with the security general scenario data set to obtain the basic sample data set.

3. The multi-task model training method according to claim 1, characterized in that The image feature extraction module includes a dynamic sequence generator and an image encoder. The dynamic sequence generator is used to process the image of each sample data in the basic sample data set to obtain a dynamic sequence; the image encoder is used to extract features from the dynamic sequence to obtain the image feature vector.

4. The multi-task model training method according to claim 3, characterized in that: The dynamic sequence generator is specifically used for: Cropping or scaling the image of each sample data in the basic sample data set to obtain an image of a fixed size; Slice and rearrange the fixed-size image to obtain an image sequence consisting of multiple sub-image blocks; Extracting information from the image sequence to obtain a binary mask of information for each sub-image block in the image sequence; Selecting sub-image blocks with high information content and sub-image blocks with low information content according to the information content binary mask of each sub-image block, and recombining the sub-image blocks with high information content and the sub-image blocks with low information content respectively to obtain a high-information image sequence and a low-information image sequence; Cross-fusing the high-information-content image sequence and the low-information-content image sequence to obtain a fused image sequence; A first position code is randomly generated, and the first position code is multiplied or added to the fused image sequence to obtain a dynamic sequence.

5. The multi-task model training method according to claim 4, characterized in that: An image information distribution network composed of a plurality of stacked multi-layer perceptrons is used to extract information from the image sequence to obtain an information binary mask for each sub-image block in the image sequence.

6. The multi-task model training method according to claim 4, characterized in that: The high-information image sequence and the low-information image sequence are cross-fused, and the specific implementation process is as follows: Using a first fully connected layer to integrate and compress the high-information image sequence to obtain a query vector of the high-information image sequence; Using a second fully connected layer to integrate and compress the low-information image sequence to obtain a key vector of the low-information image sequence; Using a third fully connected layer to integrate and compress the low-information image sequence to obtain a value vector of the low-information image sequence; Calculating a dot product between the query vector of the high-information image sequence and the key vector of the low-information image sequence, and performing a Softmax operation on the dot product result to obtain a weight matrix; The weight matrix is ​​multiplied by the value vector of the low-information image sequence to obtain a fused image sequence.

7. The multi-task model training method according to claim 3, characterized in that: The image encoder is an image encoder based on the Transformer architecture, and the image encoder is composed of a stack of L1 self-attention layers + a feedforward network.

8. The multi-task model training method according to any one of claims 1 to 7, characterized in that: The text feature extraction module is specifically used for: Performing word segmentation on the text description of each sample data in the basic sample data set to obtain a word sequence consisting of multiple words; Convert each word in the word sequence into a word vector; Randomly generate a second position code for each word vector, and multiply or add the second position code with the word vector to obtain a word sequence vector; A text encoder composed of a stack of L2 self-attention layers and a feedforward network is used to perform feature extraction on the word sequence vector to obtain the text feature vector.

9. The multi-task model training method according to any one of claims 1 to 7, characterized in that: The feature alignment module is specifically used for: Performing dimensionality transformation on the image feature vector and the text feature vector respectively so that the image feature vector and the text feature vector are mapped to the same dimension, thereby obtaining aligned image feature vector and text feature vector; Calculating the similarity between the aligned image feature vector and the text feature vector, and adjusting the parameters of the image feature extraction module and the text feature extraction module according to the matching relationship between the aligned image feature vector and the text feature vector, so as to maximize or minimize the similarity between the aligned image feature vector and the text feature vector; The aligned image feature vector and text feature vector are respectively integrated and compressed to obtain a first query vector and a second query vector, respectively; the mean square error between the first query vector and the second query vector is calculated, and the parameters of the image feature extraction module and the text feature extraction module are adjusted according to the matching relationship between the first query vector and the second query vector to maximize or minimize the mean square error between the first query vector and the second query vector.

10. The multi-task model training method according to any one of claims 1 to 7, characterized in that: The feature fusion module includes a text decoder and a multi-task head network, wherein the text decoder is used to fuse the acquired hint vector with the image feature vector to obtain image-based text features; The multi-task head network is used to perform feature extraction on the image-based text features to obtain a predicted text description.

11. The multi-task model training method according to claim 10, characterized in that: The multi-task head network adopts a large language model built based on the Transformer architecture, which is composed of L3 self-attention layers + cross-attention layers + feedforward networks stacked together.

12. The multi-task model training method according to any one of claims 1 to 7, characterized in that: The first training phase and the second training phase are performed independently of each other or alternately using a rotation strategy.

13. A multi-task prediction method, characterized in that: The prediction method comprises the following steps: Constructing a task sample data set according to the prediction task, wherein each sample data in the task sample data set includes an image and its manually annotated label; Calling a multi-task model trained by the multi-task model training method according to any one of claims 1 to 12, fine-tuning the parameters of a feature fusion module in the multi-task model using the task sample data set, while keeping the parameters of an image feature extraction module and a text feature extraction module unchanged, to obtain a target task model; Obtain an image to be predicted, and use the target task model to predict the image to be predicted to obtain a prediction result.

14. A terminal device comprising a memory, a processor, and a computer program / instruction stored in the memory, characterized in that: The processor executes the computer program / instructions to implement the method according to any one of claims 1 to 13.

15. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images